Organizations: Institute of Computing Technology, Chinese Academy of Science · School of Computer Science and Technology, University of Chinese Academy of Sciences
The Segment Anything Model 3 (SAM3) has achieved significant progress in Promptable Concept Segmentation (PCS) by processing short noun phrases to generate segmentation masks with unique instance identifiers. While SAM3 excels at single-concept segmentation, it falls short in complex, real-world environments where multiple concepts must be segmented simultaneously. In particular, SAM3 typically fails to simultaneously process concepts with semantic overlap, leading to semantic misclassifications and redundant mask predictions. Moreover, as the number of concepts increases, the independent inference paradigm introduces prohibitive inference latency. To address these limitations, we propose the Dense Competitive Representations SAM (DCR-SAM) framework. By decoupling cross-modal interactions, the proposed method alleviates the inference latency. To resolve semantic misclassifications, DCR-SAM dynamically aligns textual concepts with visual representations and incorporates a dense competition head to enforce explicit inter-class competition. Furthermore, a learnable background token is applied to absorb non-target objects dynamically. These mechanisms effectively suppress overlapping predictions and generate high-fidelity masks. Extensive experimental evaluations across multiple benchmarks demonstrate that the proposed framework: (1) achieves a 2.3% ∼ 10.7% performance improvement over state-of-the-art methods, (2) delivers a 33× inference speedup compared to the native SAM3, and (3) exhibits strong robustness across diverse scenarios.
Figures & tables
Figure 1
Figure 3: Overview of our proposed DCR-SAM framework. It transforms SAM3 from an independent decoding paradigm to a dense competitive paradigm. First, we alleviate the inference latency via Decoupled Visual Conditioning to obtain a globally contextualized visual representation. Then, we align text and visual features using the VCTR and VGTP modules. Finally, we design a competition head to enforce explicit inter-class competition and generate high-fidelity masks.
Methods
Pub. & Year
Backbone
Train on Natural Scene
ADE20K
Cityscapes
COCO-Stuff
Context59
DeepLabV3 [ 11 ]
ECCV’18
ResNet-101
42.6
77.1
37.9
52.0
DeepLabV3+ [ 12 ]
ECCV’18
ResNet-101
44.5
73.6
–
53.2
HiPoSeg [ 32 ]
ICLR’26
ResNet-101
49.0
84.0
–
–
HRNet [ 51 ]
arXiv’19
HRNetV2-W48
41.9
78.5
36.0
51.9
OCRNet [ 62 ]
ECCV’20
HRNetV2-W48
41.5
80.0
39.0
54.4
Table 1: Comparison with state-of-the-art semantic segmentation methods on natural scene and remote sensing, with the best results in bold , the second-best underlined , and all results evaluated on the validation set. † denotes Native-Binary paradigm and ‡ denotes Native-Multi-Class paradigm.
Figure 4: Qualitative comparison results on natural scene and remote sensing. We use the SAM3 † to present the visualization results of the second row.
Figure 5Figure 6Table 7
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Overview of two SAM3 baseline models for multi-class semantic segmentation . We detail the construction of the concept space and the differences in training settings between the two baselines.
Target Category Count
Semantic Space Composition
Time. ↓
Overall mIoU (%)
Base-30 mIoU (%)
N=30
Top-30 frequent classes only
236
79.4
79.4
N=60
Base-30 + Next 30 frequent classes
238
75.6 ↓ 3.8
77.5 ↓ 1.9
N=90
Base-30 + Next 60 frequent classes
241
71.5 ↓ 7.9
73.9 ↓ 5.5
N=150
Full concept space
244
68.7 ↓ 10.7
70.3 ↓ 9.1
Appendix
Table 6: Impact of the concept space scale on the ADE20K dataset.
Figure 10
Pooling Strategy
ADE20K
Cityscapes
LoveDA
Potsdam
Time. ↓
mIoU
Time. ↓
mIoU
Time. ↓
mIoU
Time. ↓
mIoU
SPA (Spatial-Preserving)
248
63.9
184
81.5
187
59.6
185
85.8
VGTP (GAP-based)
244
68.7
183
89.1
183
66.5
183
90.2
Appendix
Table 7: Ablation study on the text sequence pooling strategy across four diverse datasets.
Inference Strategy
Backbone Passes
Time (ms) ↓
Memory (MB) ↓
SAM 3 (Sequential, bs=1)
150
8038
9010
SAM 3 (Batched, bs=15)
10
6392
12960
SAM 3 (Batched, bs=25)
6
6067
17488
SAM 3 (Batched, bs=30)
5
5966
19816
SAM 3 (Batched, bs=50)
3
5687
29080
DCR-SAM (Ours)
1
244
6318
Appendix
Table 8: Efficiency comparison between SAM 3 and DCR-SAM across different batch sizes.
Figure 10: Cross-modal semantic activation maps. Aggregation in the original SAM3 leads to overlapping masks, whereas our method mitigates this issue through explicit inter-class competition.
Figure 11: Qualitative comparison results on ADE20k dataset.
Figure 12: Qualitative comparison results on Cityscapes dataset.
Figure 13: Qualitative comparison results on COCO-Stuff dataset.
Figure 14: Qualitative comparison results on Context59 dataset.
Figure 15: Qualitative comparison results on LoveDA dataset.
Figure 16: Qualitative comparison results on Potsdam dataset.
Figure 17: Qualitative comparison results on Vaihingen dataset.
Figure 18: Qualitative comparison results on iSAID dataset.
SAM3 advances open-vocabulary semantic segmentation by introducing a prompt-driven mask generation paradigm. However, in multi-class open-vocabulary scenarios, masks generated independently from different category prompts lack a unified and inter-class comparable evidence scale, often resulting in overlapping coverage and unstable competition. Moreover, synonymous expressions of the same concept tend to activate inconsistent semantic and spatial evidence, leading to intra-class drift that exacerbates inter-class conflicts and compromises overall inference stability. To address these issues, we propose CoCo-SAM3 (Concept-Conflict SAM3), which explicitly decouples inference into intra-class enhancement and inter-class competition. Our method first aligns and aggregates evidence from synonymous prompts to strengthen concept consistency. It then performs inter-class competition on a unified comparable scale, enabling direct pixel-wise comparisons among all candidate classes. This mechanism stabilizes multi-class inference and effectively mitigates inter-class conflicts. Without requiring any additional training, CoCo-SAM3 achieves consistent improvements across eight open-vocabulary semantic segmentation benchmarks.
Yanhui Chen, Baoyao Yang, Siqi Liu +1
School of Computers, Guangdong University of Technology · Shenzhen Research Institute of Big Data · Peking University
The recent Segment Anything Model 3 (SAM3) extends powerful promptable segmentation from spatial cues to concepts, raising the prospect of a single foundation model for universal image segmentation. However, this promise often fails to translate into reliable performance on downstream segmentation tasks. In this work, we empirically identify concept recognition, rather than mask quality, as the primary bottleneck: SAM3 accurately delineates the objects it recognizes yet struggles to robustly determine which concepts are present. Driven by this, we present Text Anchor re-Grounding (\TAG{}), a simple yet effective framework that rectifies the concept recognition of SAM3 on the target domain without dense annotations. Specifically, \TAG{} introduces an Anchor Residual Pathway comprising image-conditioned, low-rank adapters to re-ground text anchors in the target distribution and modulate subsequent text-conditioned interactions. Guided solely by image-level category labels, \TAG{} jointly optimizes binary cross-entropy with Soft Boundary Ranking objectives to calibrate presence confidence and distinguish present concepts from confusable negatives. Building upon this, we devise Confidence-Weighted Distillation to couple rich semantic context of a multimodal large language model with fine-grained spatial priors of SAM3, thereby achieving effective calibration in the absence of manual annotations. Extensive experiments demonstrate that \TAG{} unlocks SAM3's segmentation potential with only 3.1M trainable parameters, achieving impressive performance across diverse scenarios, and notably, even surpassing leading mask-supervised methods on several metrics. Our code will be publicly available at GitHub \faGithub.
We propose Semantic-Fast-SAM (SFS), a semantic segmentation framework that combines the Fast Segment Anything model with a semantic labeling pipeline to achieve real-time performance without sacrificing accuracy. FastSAM is an efficient CNN-based re-implementation of the Segment Anything Model (SAM) that runs much faster than the original transformer-based SAM. Building upon FastSAM's rapid mask generation, we integrate a Semantic-Segment-Anything (SSA) labeling strategy to assign meaningful categories to each mask. The resulting SFS model produces high-quality semantic segmentation maps at a fraction of the computational cost and memory footprint of the original SAM-based approach. Experiments on Cityscapes and ADE20K benchmarks demonstrate that SFS matches the accuracy of prior SAM-based methods (mIoU ~ 70.33 on Cityscapes and 48.01 on ADE20K) while achieving approximately 20x faster inference than SSA in the closed-set setting. We also show that SFS effectively handles open-vocabulary segmentation by leveraging CLIP-based semantic heads, outperforming recent open-vocabulary models on broad class labeling. This work enables practical real-time semantic segmentation with the "segment-anything" capability, broadening the applicability of foundation segmentation models in robotics scenarios. The implementation is available at https://github.com/KBH00/Semantic-Fast-SAM.