UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

Junwei Yu; Trevor Darrell; XuDong Wang

cs.CVNov 17, 2025

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

Authors: Junwei Yu, Trevor Darrell, XuDong Wang

Organizations: UC Berkeley

Abstract

The Segment Anything Model (SAM) family has become a widely adopted vision foundation model, but its ability to control segmentation granularity remains limited. Users often need to refine results manually - by adding more prompts or selecting from pre-generated masks - to achieve the desired level of detail. This process can be ambiguous, as the same prompt may correspond to several plausible masks, and collecting dense annotations across all granularities is prohibitively expensive, making supervised solutions infeasible. To address this limitation, we introduce UnSAMv2, which can segment anything at any granularity without using any human annotations. UnSAMv2 extends the divide-and-conquer strategy of UnSAM by discovering abundant mask-granularity pairs and introducing a novel granularity control embedding that enables precise, continuous control over segmentation scale. Remarkably, with only $6$ K unlabeled images and $0.02\%$ additional parameters, UnSAMv2 substantially enhances SAM-2, enabling segmentation of anything at any granularity across interactive, whole-image, and video segmentation tasks. Furthermore, we introduce a language-guided segmentation pipeline that treats granularity as a predictable variable by grounding it in text semantics via an external vision-language model, improving upon the previous state of the art in 1-IoU (40.4 $\rightarrow$ 47.3). Across 11+ benchmarks, UnSAMv2 achieves consistent gains over SAM/SAM-2 baselines, improving $\text{NoC}_{90}$ (5.69 $\rightarrow$ 4.75), 1-IoU (58.0 $\rightarrow$ 73.1), and $\text{AR}_{1000}$ (49.6 $\rightarrow$ 68.3) by a large margin. Our work demonstrates that a small amount of unlabeled data can deliver large, practical improvements to strong supervised baselines. Our code, dataset, and demo are available at https://github.com/yujunwei04/UnSAMv2.

Figures & tables

Figure 1: What is an object? The notion has long been debated: should it follow a model’s learned semantics or an annotator’s subjective judgment? UnSAMv2 takes a third path, granting users full flexibility to define objectness through promptable segmentation with a single point and a continuous, controllable granularity score. Built on SAM-2 ( Ravi et al., 2024 ) , UnSAMv2 introduces a granularity-aware, self-supervised training pipeline based on divide-and-conquer pseudo-labels ( Wang et al., 2024c ) . Trained on just 6000 unlabeled images, it segments anything from fine-grained parts to holistic objects, achieving state-of-the-art performance across interactive, whole-image, and video segmentation tasks.

Panel of Figure 2 — Figure 2: From ambiguity to control, and learning across scales. (a) Without granularity input, SAM-2 may output multiple plausible masks per point; UnSAMv2 introduces a continuous granularity variable to retrieve the intended object at any scale with a single prompt. (b) Our unsupervised divide-and-conquer pipeline yields a rich, left-tailed granularity distribution dominated by fine-grained pseudo-masks, yet UnSAMv2 learns stable semantics across all scales.

Figure 3: Granularity is a relative notion and robust to camera distance. At a fixed granularity value, mask sizes vary widely across scenes, showing that UnSAMv2 learns granularity relationally, consistent with human perception of parts and wholes rather than simply associating it with absolute size.

Figure 4: Architecture of UnSAMv2 . Built on SAM-2, UnSAMv2 introduces a Fourier-based granularity encoder and a mask token to enable segmentation at any granularity. A scalar granularity input $g\!\in\![0.1,1]$ is mapped to a high-dimensional embedding via Fourier transformation and an MLP, then injected into the transformer alongside the sparse point embedding and dense image embedding. The granularity-aware mask token attends to image, point, and granularity embeddings, and is finally decoded by a token decoder into a mask at the requested granularity.

Figure 5: Quantitative analysis on granularity and absolute mask areas.

Figure 6: Qualitative comparison with the previous state-of-the-art method ( Zhao et al., 2024 ) . Each scene shows results at different target granularity values. Prior methods often break one object into parts at high granularity or include extra regions at low granularity. In contrast, UnSAMv2 produces clear and consistent masks with smooth transitions across scales.

Figure 7: Language-guided segmentation. Conditioned on the image and the text description, the VLM infers a granularity value, which is then provided to UnSAMv2 for segmentation.

Datasets $\rightarrow$	Averaged			GrabCut			Berkeley			SBD			PascalPart			PartImageNet
Method	$\text{NoC}_{80\downarrow}$	$\text{NoC}_{90\downarrow}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{90}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{90}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{90}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{85}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{85}$	1-IoU
SAM-2 ( Ravi et al., 2024 )	4.02	5.69	58.0	1.48	1.54	76.8	1.29	1.72	80.0	2.85	6.73	64.6	9.37	11.80	29.0	5.09	6.64	39.5
UnSAMv2 ∗	3.41	4.75	73.1	1.16	1.30	91.4	1.10	1.60	90.8	2.75	5.97	74.8	7.84	9.59	51.5	4.21	5.29	57.1
vs. sup. SAM-2	-0.61	-0.94	+15.1	-0.32	-0.24	+14.6	-0.19	-0.12	+10.8	-0.10	-0.76	+10.2	-1.53	-2.21	+22.5	-0.88	-1.35	+17.6
UnSAMv2+ ∗	3.30	4.56	74.7	1.22	1.26	92.6	1.12	1.37	91.1	2.55	5.72	77.3	7.87	9.67	52.0	3.75	4.77	60.7

Table 1: Comparison with SAM-2 on interactive segmentation benchmarks. Trained on 6,000 images with purely unsupervised pseudo-labels, UnSAMv2 significantly outperforms SAM-2. We report promptable segmentation performance in terms of NoC and 1-IoU across five benchmarks. * Following ( Zhao et al., 2024 ) , we select the optimal granularity from 0.1 to 1.0 in steps of 0.1 and report averaged results.

Datasets $\rightarrow$	Averaged			GrabCut			Berkeley			DAVIS			SA-1B			PartImageNet
Method	$\text{NoC}_{80\downarrow}$	$\text{NoC}_{90\downarrow}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{90}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{90}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{90}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{90}$	1-IoU	$\text{NoC}_{80}$	$\text{NoC}_{85}$	1-IoU
SimpleClick ( Liu et al., 2023 )	3.32	4.87	60.2	1.32	1.48	89.6	1.26	2.44	84.6	2.88	5.38	75.6	4.80	7.27	21.9	6.34	7.76	29.2
GraCo ( Zhao et al., 2024 ) ∗	2.35	3.42	74.4	1.24	1.32	91.1	1.17	1.63	88.7	2.84	5.09	74.0	2.41	4.01	66.2	4.11	5.03	51.7
SAM-2 ( Ravi et al., 2024 )	2.44	3.63	69.0	1.48	1.54	76.8	1.29	1.72	80.0	2.51	4.50	71.7	1.82	3.76	77.1	5.09	6.64	39.5
UnSAMv2 ∗	2.28	3.40	79.3	1.16	1.30	91.4	1.10	1.60	90.8	2.75	4.66	84.7	2.18	4.16	72.4	4.21	5.29	57.1
UnSAMv2+ ∗	2.07	3.10	81.7	1.22	1.26	92.6	1.12	1.37	91.1	2.36	4.40	85.7	1.87	3.69	78.4	3.75	4.77	60.7

Table 2: State-of-the-art performance on interactive segmentation across benchmarks. Trained with only 6,000 images using a combination of supervised and unsupervised labels, UnSAMv2+ demonstrates superior segmentation quality and indicates how effectively unsupervised methods can complement supervised data. Results of ( Liu et al., 2023 ; Zhao et al., 2024 ) are reproduced with official code and checkpoints. *: Following ( Zhao et al., 2024 ) , we select optimal granularity from 0.1 to 1.0 with a step of 0.1 and report average results.

Table 10

		UnSAMv2
Dataset	SAM-2	$g^{*}$	$\delta{=}0.05$	$\delta{=}0.10$
PascalPart	9.37	7.84	8.39	8.86
PartImageNet	5.09	4.21	4.57	4.92
Berkeley	1.29	1.10	1.16	1.28
GrabCut	1.48	1.16	1.28	1.34

Table 5: Robustness to imperfect granularity. NoC 80 (

\downarrow

) with the granularity off the oracle

g^{*}

\delta

Method	NoC ${}_{80}\downarrow$	1-IoU
SAM-2 (3 tok.)	4.02	58.0
UnSAMv2 (3 gra.)	3.76	66.7
UnSAMv2 (10 gra.)	3.41	73.1

Table 6: Evaluation with three granularity scales.

Figure 8: Visualizations for whole-image segmentation. Low granularity reveals fine parts, while higher values recover whole objects. UnSAMv2 offers controllable whole-image segmentation, even for scenes with many densely packed entities.

Method	J&F	Anch. J&F	TCE $\downarrow$
SAM-2	57.6	47.0	11.15
UnSAMv2	64.2	50.3	10.88

Table 7: Video results on DAVIS.

Figure 10: Ablation of granularity-aware mask token. Directly finetuning original SAM-2 mask tokens leads to limited gains, suggesting they already encode strong priors. Adding our granularity-aware token enables efficient learning of granularity.

Table 17

VLM	Input	Spearman
Qwen3-VL-8B	Image + text	+0.81
Qwen3-VL-8B	Text only	+0.75

Table 12: Semantic alignment between VLM-predicted and oracle granularity (PartImageNet test).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Property	UnSAM	SOHES	GraCo	SAM	UnSAMv2
Unsupervised	✓	✓	✗	✗	✓
Granularity input	✗	✗	✓	✗	✓
Continuous granularity	✗	✗	✗	✗	✓
Hierarchical perspective	✓	✗	✗	✗	✓

Appendix

Table A1 : Technical comparison with prior work (UnSAM ( Wang et al., 2024c ) , SOHES ( Cao et al., 2024 ) , GraCo ( Zhao et al., 2024 ) , SAM ( Kirillov et al., 2023 ) ). UnSAMv2 is the first to learn a continuous granularity variable from unsupervised part–whole hierarchies.

Model (# output masks)	PascalPart	PtIn-OOD	PACO	PtIn
Semantic-SAM ( Li et al., 2023a ) (6)	46.7	71.0	54.2	48.5
UnSAMv2 (6 $g$ )	48.4	72.5	55.1	51.2
UnSAMv2+ (6 $g$ )	50.6	73.8	59.2	54.9
UnSAMv2+ (10 $g$ )	55.8	76.1	64.3	60.7

Appendix

Table A2 : Comparison with Semantic-SAM ( Li et al., 2023a ) . We report per-target best IoU over each model’s output set with identical clicks on PascalPart, PartImageNet-OOD / PartImageNet, and PACO. UnSAMv2 outperforms Semantic-SAM on all evaluated benchmarks while learning granularity control from unlabeled data.

Model	Masks generated	w/ conf. filtering + NMS
SAM-2 (3 mask tokens per grid point)	635	–
UnSAMv2 (10 $g$ values per grid point)	1,206	714

Appendix

Table A3 : Whole-image proposal counts per image. We prompt UnSAMv2 with a

64{\times}64

point grid at each

g\in\{0.1,0.2,\ldots,1.0\}

, keep masks with confidence score

>0.5

, and deduplicate with NMS at IoU threshold

0.7

. Filtering and deduplication reduce the pool from over 1,200 to about 700 masks, enabling a fair recall comparison with the baseline.

Figure A1 : Interactive segmentation results on SA-1B. The top row is previous SOTA GraCo ( Zhao et al., 2024 ) and the bottom row is UnSAMv2 .

Figure A3 : UnSAMv2 ’s interactive video segmentation results on YoutubeVIS dataset at various granularity levels.

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 390 papers

Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision
Sep 30, 2026Weijian Jian, Xiaoyue Zhang, Bin Xiao +5Image Segmentation Self-Supervised Learning
AI Research · Independent Researcher · University of Ottawa +1
TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision
Oct 10, 2026Hao Zhu, Heyu Sun, Shuo Jin +2Promptable Image Segmentation Open-Vocabulary Semantic Segmentation
X2SAM: Any Segmentation in Images and Videos
Apr 27, 2026Hao Wang, Limeng Qiao, Chi Zhang +4Image Segmentation Video Object Segmentation
Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc

Sep 30, 2026J/K move · Enter open · S save

Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

Weijian Jian, Xiaoyue Zhang, Bin Xiao, Chunyu Xie+4

AI Research · Independent Researcher · University of Ottawa+1

Image Segmentation Self-Supervised Learning Promptable Image Segmentation Unsupervised Image Segmentation

Open paper AskPDF arXiv

The Segment Anything Model (SAM) relies heavily on massive manual annotations, creating a fundamental bottleneck for model scaling. While unsupervised methods attempt to learn object concepts from motion, they typically overfit to moving entities, lacking both multi-granularity understanding and the ability to generalize to static objects. To overcome this, we introduce Motion-Grounded Segment Anything (MoSA), a highly scalable unsupervised framework that learns a transferable objectness prior from unlabeled videos. MoSA operates in three progressive stages: (1) automatically generating multi-granularity motion pseudo-labels from large-scale video data; (2) training a Perceptual Grouping Model (PGM) via contrastive learning to internalize a generalized, appearance-driven concept of objects; and (3) transferring this learned prior into a prompt-guided architecture for segment-anything-style inference on images. Extensive zero-shot evaluations across seven challenging benchmarks (e.g., COCO and ADE20K) demonstrate that MoSA significantly outperforms existing unsupervised methods. Notably, despite using zero manual annotations, MoSA achieves segmentation performance comparable to the fully supervised SAM. Our findings reveal that harnessing large-scale unlabeled motion is a feasible and highly scalable alternative to annotation-driven segment-anything pipelines.

Abstract

Image Segmentation

Explore similar work

Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision

X2SAM: Any Segmentation in Images and Videos