The Segment Anything Model (SAM) family has become a widely adopted vision foundation model, but its ability to control segmentation granularity remains limited. Users often need to refine results manually - by adding more prompts or selecting from pre-generated masks - to achieve the desired level of detail. This process can be ambiguous, as the same prompt may correspond to several plausible masks, and collecting dense annotations across all granularities is prohibitively expensive, making supervised solutions infeasible. To address this limitation, we introduce UnSAMv2, which can segment anything at any granularity without using any human annotations. UnSAMv2 extends the divide-and-conquer strategy of UnSAM by discovering abundant mask-granularity pairs and introducing a novel granularity control embedding that enables precise, continuous control over segmentation scale. Remarkably, with only 6K unlabeled images and 0.02% additional parameters, UnSAMv2 substantially enhances SAM-2, enabling segmentation of anything at any granularity across interactive, whole-image, and video segmentation tasks. Furthermore, we introduce a language-guided segmentation pipeline that treats granularity as a predictable variable by grounding it in text semantics via an external vision-language model, improving upon the previous state of the art in 1-IoU (40.4 → 47.3). Across 11+ benchmarks, UnSAMv2 achieves consistent gains over SAM/SAM-2 baselines, improving NoC90 (5.69 → 4.75), 1-IoU (58.0 → 73.1), and AR1000 (49.6 → 68.3) by a large margin. Our work demonstrates that a small amount of unlabeled data can deliver large, practical improvements to strong supervised baselines. Our code, dataset, and demo are available at https://github.com/yujunwei04/UnSAMv2.
Figures & tables
Figure 1: What is an object? The notion has long been debated: should it follow a model’s learned semantics or an annotator’s subjective judgment? UnSAMv2 takes a third path, granting users full flexibility to define objectness through promptable segmentation with a single point and a continuous, controllable granularity score. Built on SAM-2 ( Ravi et al., 2024 ) , UnSAMv2 introduces a granularity-aware, self-supervised training pipeline based on divide-and-conquer pseudo-labels ( Wang et al., 2024c ) . Trained on just 6000 unlabeled images, it segments anything from fine-grained parts to holistic objects, achieving state-of-the-art performance across interactive, whole-image, and video segmentation tasks.
Figure 2: From ambiguity to control, and learning across scales. (a) Without granularity input, SAM-2 may output multiple plausible masks per point; UnSAMv2 introduces a continuous granularity variable to retrieve the intended object at any scale with a single prompt. (b) Our unsupervised divide-and-conquer pipeline yields a rich, left-tailed granularity distribution dominated by fine-grained pseudo-masks, yet UnSAMv2 learns stable semantics across all scales.
Figure 3: Granularity is a relative notion and robust to camera distance. At a fixed granularity value, mask sizes vary widely across scenes, showing that UnSAMv2 learns granularity relationally, consistent with human perception of parts and wholes rather than simply associating it with absolute size.
Figure 4: Architecture of UnSAMv2 . Built on SAM-2, UnSAMv2 introduces a Fourier-based granularity encoder and a mask token to enable segmentation at any granularity. A scalar granularity input g∈[0.1,1] is mapped to a high-dimensional embedding via Fourier transformation and an MLP, then injected into the transformer alongside the sparse point embedding and dense image embedding. The granularity-aware mask token attends to image, point, and granularity embeddings, and is finally decoded by a token decoder into a mask at the requested granularity.
Figure 5: Quantitative analysis on granularity and absolute mask areas.
Figure 6: Qualitative comparison with the previous state-of-the-art method ( Zhao et al., 2024 ) . Each scene shows results at different target granularity values. Prior methods often break one object into parts at high granularity or include extra regions at low granularity. In contrast, UnSAMv2 produces clear and consistent masks with smooth transitions across scales.
Figure 7: Language-guided segmentation. Conditioned on the image and the text description, the VLM infers a granularity value, which is then provided to UnSAMv2 for segmentation.
Datasets →
Averaged
GrabCut
Berkeley
SBD
PascalPart
PartImageNet
Method
NoC80↓
NoC90↓
1-IoU
NoC80
NoC90
1-IoU
NoC80
NoC90
1-IoU
NoC80
NoC90
1-IoU
NoC80
NoC85
1-IoU
NoC80
NoC85
1-IoU
SAM-2 ( Ravi et al., 2024 )
4.02
5.69
58.0
1.48
1.54
76.8
1.29
1.72
80.0
2.85
6.73
64.6
9.37
11.80
29.0
5.09
6.64
39.5
UnSAMv2 ∗
3.41
4.75
73.1
1.16
1.30
91.4
1.10
1.60
90.8
2.75
5.97
74.8
7.84
9.59
51.5
4.21
5.29
57.1
vs. sup. SAM-2
-0.61
-0.94
+15.1
-0.32
-0.24
+14.6
-0.19
-0.12
+10.8
-0.10
-0.76
+10.2
-1.53
-2.21
+22.5
-0.88
-1.35
+17.6
UnSAMv2+ ∗
3.30
4.56
74.7
1.22
1.26
92.6
1.12
1.37
91.1
2.55
5.72
77.3
7.87
9.67
52.0
3.75
4.77
60.7
Table 1: Comparison with SAM-2 on interactive segmentation benchmarks. Trained on 6,000 images with purely unsupervised pseudo-labels, UnSAMv2 significantly outperforms SAM-2. We report promptable segmentation performance in terms of NoC and 1-IoU across five benchmarks. * Following ( Zhao et al., 2024 ) , we select the optimal granularity from 0.1 to 1.0 in steps of 0.1 and report averaged results.
Datasets →
Averaged
GrabCut
Berkeley
DAVIS
SA-1B
PartImageNet
Method
NoC80↓
NoC90↓
1-IoU
NoC80
NoC90
1-IoU
NoC80
NoC90
1-IoU
NoC80
NoC90
1-IoU
NoC80
NoC90
1-IoU
NoC80
NoC85
1-IoU
SimpleClick ( Liu et al., 2023 )
3.32
4.87
60.2
1.32
1.48
89.6
1.26
2.44
84.6
2.88
5.38
75.6
4.80
7.27
21.9
6.34
7.76
29.2
GraCo ( Zhao et al., 2024 ) ∗
2.35
3.42
74.4
1.24
1.32
91.1
1.17
1.63
88.7
2.84
5.09
74.0
2.41
4.01
66.2
4.11
5.03
51.7
SAM-2 ( Ravi et al., 2024 )
2.44
3.63
69.0
1.48
1.54
76.8
1.29
1.72
80.0
2.51
4.50
71.7
1.82
3.76
77.1
5.09
6.64
39.5
UnSAMv2 ∗
2.28
3.40
79.3
1.16
1.30
91.4
1.10
1.60
90.8
2.75
4.66
84.7
2.18
4.16
72.4
4.21
5.29
57.1
UnSAMv2+ ∗
2.07
3.10
81.7
1.22
1.26
92.6
1.12
1.37
91.1
2.36
4.40
85.7
1.87
3.69
78.4
3.75
4.77
60.7
Table 2: State-of-the-art performance on interactive segmentation across benchmarks. Trained with only 6,000 images using a combination of supervised and unsupervised labels, UnSAMv2+ demonstrates superior segmentation quality and indicates how effectively unsupervised methods can complement supervised data. Results of ( Liu et al., 2023 ; Zhao et al., 2024 ) are reproduced with official code and checkpoints. *: Following ( Zhao et al., 2024 ) , we select optimal granularity from 0.1 to 1.0 with a step of 0.1 and report average results.
Table 10
UnSAMv2
Dataset
SAM-2
g∗
δ=0.05
δ=0.10
PascalPart
9.37
7.84
8.39
8.86
PartImageNet
5.09
4.21
4.57
4.92
Berkeley
1.29
1.10
1.16
1.28
GrabCut
1.48
1.16
1.28
1.34
Table 5: Robustness to imperfect granularity. NoC 80 ( ↓ ) with the granularity off the oracle g∗ by δ .
Method
NoC 80↓
1-IoU
SAM-2 (3 tok.)
4.02
58.0
UnSAMv2 (3 gra.)
3.76
66.7
UnSAMv2 (10 gra.)
3.41
73.1
Table 6: Evaluation with three granularity scales.
Figure 8: Visualizations for whole-image segmentation. Low granularity reveals fine parts, while higher values recover whole objects. UnSAMv2 offers controllable whole-image segmentation, even for scenes with many densely packed entities.
Method
J&F
Anch. J&F
TCE ↓
SAM-2
57.6
47.0
11.15
UnSAMv2
64.2
50.3
10.88
Table 7: Video results on DAVIS.
Figure 15
Figure 10: Ablation of granularity-aware mask token. Directly finetuning original SAM-2 mask tokens leads to limited gains, suggesting they already encode strong priors. Adding our granularity-aware token enables efficient learning of granularity.
Table 17
VLM
Input
Spearman
Qwen3-VL-8B
Image + text
+0.81
Qwen3-VL-8B
Text only
+0.75
Table 12: Semantic alignment between VLM-predicted and oracle granularity (PartImageNet test).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Property
UnSAM
SOHES
GraCo
SAM
UnSAMv2
Unsupervised
✓
✓
✗
✗
✓
Granularity input
✗
✗
✓
✗
✓
Continuous granularity
✗
✗
✗
✗
✓
Hierarchical perspective
✓
✗
✗
✗
✓
Appendix
Table A1 : Technical comparison with prior work (UnSAM ( Wang et al., 2024c ) , SOHES ( Cao et al., 2024 ) , GraCo ( Zhao et al., 2024 ) , SAM ( Kirillov et al., 2023 ) ). UnSAMv2 is the first to learn a continuous granularity variable from unsupervised part–whole hierarchies.
Model (# output masks)
PascalPart
PtIn-OOD
PACO
PtIn
Semantic-SAM ( Li et al., 2023a ) (6)
46.7
71.0
54.2
48.5
UnSAMv2 (6 g )
48.4
72.5
55.1
51.2
UnSAMv2+ (6 g )
50.6
73.8
59.2
54.9
UnSAMv2+ (10 g )
55.8
76.1
64.3
60.7
Appendix
Table A2 : Comparison with Semantic-SAM ( Li et al., 2023a ) . We report per-target best IoU over each model’s output set with identical clicks on PascalPart, PartImageNet-OOD / PartImageNet, and PACO. UnSAMv2 outperforms Semantic-SAM on all evaluated benchmarks while learning granularity control from unlabeled data.
Model
Masks generated
w/ conf. filtering + NMS
SAM-2 (3 mask tokens per grid point)
635
–
UnSAMv2 (10 g values per grid point)
1,206
714
Appendix
Table A3 : Whole-image proposal counts per image. We prompt UnSAMv2 with a 64×64 point grid at each g∈{0.1,0.2,…,1.0} , keep masks with confidence score >0.5 , and deduplicate with NMS at IoU threshold 0.7 . Filtering and deduplication reduce the pool from over 1,200 to about 700 masks, enabling a fair recall comparison with the baseline.
Figure A1 : Interactive segmentation results on SA-1B. The top row is previous SOTA GraCo ( Zhao et al., 2024 ) and the bottom row is UnSAMv2 .
Figure A2 : Whole image segmentation on SA-1B. From top to bottom are raw images, segmentation by SAM-2 and UnSAMv2 .
Figure A3 : UnSAMv2 ’s interactive video segmentation results on YoutubeVIS dataset at various granularity levels.
The Segment Anything Model (SAM) relies heavily on massive manual annotations, creating a fundamental bottleneck for model scaling. While unsupervised methods attempt to learn object concepts from motion, they typically overfit to moving entities, lacking both multi-granularity understanding and the ability to generalize to static objects. To overcome this, we introduce Motion-Grounded Segment Anything (MoSA), a highly scalable unsupervised framework that learns a transferable objectness prior from unlabeled videos. MoSA operates in three progressive stages: (1) automatically generating multi-granularity motion pseudo-labels from large-scale video data; (2) training a Perceptual Grouping Model (PGM) via contrastive learning to internalize a generalized, appearance-driven concept of objects; and (3) transferring this learned prior into a prompt-guided architecture for segment-anything-style inference on images. Extensive zero-shot evaluations across seven challenging benchmarks (e.g., COCO and ADE20K) demonstrate that MoSA significantly outperforms existing unsupervised methods. Notably, despite using zero manual annotations, MoSA achieves segmentation performance comparable to the fully supervised SAM. Our findings reveal that harnessing large-scale unlabeled motion is a feasible and highly scalable alternative to annotation-driven segment-anything pipelines.
Weijian Jian, Xiaoyue Zhang, Bin Xiao +5
AI Research · Independent Researcher · University of Ottawa +1
The recent Segment Anything Model 3 (SAM3) extends powerful promptable segmentation from spatial cues to concepts, raising the prospect of a single foundation model for universal image segmentation. However, this promise often fails to translate into reliable performance on downstream segmentation tasks. In this work, we empirically identify concept recognition, rather than mask quality, as the primary bottleneck: SAM3 accurately delineates the objects it recognizes yet struggles to robustly determine which concepts are present. Driven by this, we present Text Anchor re-Grounding (\TAG{}), a simple yet effective framework that rectifies the concept recognition of SAM3 on the target domain without dense annotations. Specifically, \TAG{} introduces an Anchor Residual Pathway comprising image-conditioned, low-rank adapters to re-ground text anchors in the target distribution and modulate subsequent text-conditioned interactions. Guided solely by image-level category labels, \TAG{} jointly optimizes binary cross-entropy with Soft Boundary Ranking objectives to calibrate presence confidence and distinguish present concepts from confusable negatives. Building upon this, we devise Confidence-Weighted Distillation to couple rich semantic context of a multimodal large language model with fine-grained spatial priors of SAM3, thereby achieving effective calibration in the absence of manual annotations. Extensive experiments demonstrate that \TAG{} unlocks SAM3's segmentation potential with only 3.1M trainable parameters, achieving impressive performance across diverse scenarios, and notably, even surpassing leading mask-supervised methods on several metrics. Our code will be publicly available at GitHub \faGithub.
Multimodal Large Language Models (MLLMs) have demonstrated strong image-level visual understanding and reasoning, yet their pixel-level perception across both images and videos remains limited. Foundation segmentation models such as the SAM series produce high-quality masks, but they rely on low-level visual prompts and cannot natively interpret complex conversational instructions. Existing segmentation MLLMs narrow this gap, but are usually specialized for either images or videos and rarely support both textual and visual prompts in one interface. We introduce X2SAM, a unified segmentation MLLM that extends any-segmentation capabilities from images to videos. Given conversational instructions and visual prompts, X2SAM couples an LLM with a Mask Memory module that stores guided vision features for temporally consistent video mask generation. The same formulation supports generic, open-vocabulary, referring, reasoning, grounded conversation generation, interactive, and visual grounded segmentation across image and video inputs. We further introduce the Video Visual Grounded (V-VGD) segmentation benchmark, which evaluates whether a model can segment object tracks in videos from interactive visual prompts. With a unified joint training strategy over heterogeneous image and video datasets, X2SAM delivers strong video segmentation performance, remains competitive on image segmentation benchmarks, and preserves general image and video chat ability.
Hao Wang, Limeng Qiao, Chi Zhang +4
Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc