Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

Weijian Jian; Xiaoyue Zhang; Bin Xiao; Chunyu Xie; Yixiao He; Yutao Liu; Dawei Leng; Yuhui Yin

cs.CVSep 30, 2026

Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

Authors: Weijian Jian, Xiaoyue Zhang, Bin Xiao, Chunyu Xie, Yixiao He, Yutao Liu, Dawei Leng, Yuhui Yin

Organizations: AI Research · Independent Researcher · University of Ottawa · Beijing University of Posts and Telecommunications

Abstract

The Segment Anything Model (SAM) relies heavily on massive manual annotations, creating a fundamental bottleneck for model scaling. While unsupervised methods attempt to learn object concepts from motion, they typically overfit to moving entities, lacking both multi-granularity understanding and the ability to generalize to static objects. To overcome this, we introduce Motion-Grounded Segment Anything (MoSA), a highly scalable unsupervised framework that learns a transferable objectness prior from unlabeled videos. MoSA operates in three progressive stages: (1) automatically generating multi-granularity motion pseudo-labels from large-scale video data; (2) training a Perceptual Grouping Model (PGM) via contrastive learning to internalize a generalized, appearance-driven concept of objects; and (3) transferring this learned prior into a prompt-guided architecture for segment-anything-style inference on images. Extensive zero-shot evaluations across seven challenging benchmarks (e.g., COCO and ADE20K) demonstrate that MoSA significantly outperforms existing unsupervised methods. Notably, despite using zero manual annotations, MoSA achieves segmentation performance comparable to the fully supervised SAM. Our findings reveal that harnessing large-scale unlabeled motion is a feasible and highly scalable alternative to annotation-driven segment-anything pipelines.

Figures & tables

Figure 1 : Overview of our proposed MoSA . The top-left and top-right panels respectively show qualitative comparisons of MoSA against the fully supervised SAM [ 15 ] and the unsupervised UnSAM [ 38 ] on point-based promptable segmentation and whole-image segmentation. The middle-right panel shows that MoSA achieves SOTA performance compared to other unsupervised segmentation methods. The bottom panel illustrates the pipeline of MoSA’s three-stage unsupervised framework. Note that UnSAM is not strictly unsupervised, as it utilizes the supervised CascadePSP [ 7 ] refiner.

Figure 2 : An overview of our Multi-Granularity Motion Segmentation (MGMS) pipeline for generating high-quality pseudo-labels from unlabeled videos. The pipeline first extracts bidirectional optical flow from videos and processes it through two parallel modules: the Rigid Object Segmentation Module and the Motion Foreground Segmentation Module. These generate masks for rigid parts and the moving foreground of the central frame (highlighted in red). Finally, masks are filtered and processed with NMS to yield the final multi-granularity pseudo-labels.

Figure 3 : Illustration of Perceptual Grouping Contrastive Learning (PGCL). Our PGM features multiple heads, each trained on pseudo-masks of a specific scale range. For each assigned mask, we compute a similarity map by comparing an in-mask anchor patch ( + ) with all patches in the corresponding head’s feature space. The PGCL loss supervises this map using the mask as the ground truth, training the model to group coherent patches into objects.

Methods	PtIn	LVIS	Entity	PACO	ADE	COCO	SA-1B	Average
UnSAM [ 38 ] (CRF [ 18 ] )	23.2	16.5	22.1	10.8	15.1	21.9	15.3	17.8
UnSAM [ 38 ] (CascadePSP † [ 7 ] )	27.3	19.9	24.3	12.8	17.8	24.8	24.7	21.7
PGM ( ours ) (w/o refinement)	34.7	27.9	28.2	16.1	25.5	32.6	36.6	28.8
PGM ( ours ) (CRF [ 18 ] )	36.1	29.3	30.1	16.9	26.7	33.8	37.8	30.1

Table 1 : Evaluation results of PGM on seven datasets. † : The pseudo-labels used by UnSAM [ 38 ] are not strictly unsupervised as they are refined by CascadePSP [ 7 ] .

Figure 4 : Qualitative comparison of PGM (ours) with DINOv3 family [ 32 ] , where the red + denotes anchor patches.

Methods	Backbone	#	Avg.	with Whole Entities					with Parts
Methods	#params	images	Avg.	COCO	LVIS	ADE	Entity	SA-1B	PtIn	PACO
SAM [ 15 ]	VIT-B(85M)	11M	42.1	49.6	46.1	45.8	45.9	60.8	28.3	18.1
FreeSOLO [ 34 ]	RN-101 (45M)	1.3M	7.3	11.6	5.9	7.3	8.0	2.2	13.8	2.4
CutLER [ 36 ]	RN-50 (23M)	1.3M	21.8	28.1	20.2	26.3	23.1	17.0	28.7	8.9
SOHES [ 3 ]	VIT-B(85M)	0.2M	30.1	30.5	29.1	31.1	33.5	33.3	36.0	17.1
UnSAM † [ 38 ]	RN-50(23M)	0.1M	39.2	40.5	37.7	35.7	39.6	41.9	51.6	27.5

Table 2 : Results on the whole-image segmentation task. † : UnSAM [ 38 ] utilizes CascadePSP [ 7 ] , a method that relies on manual annotations, and is therefore not strictly unsupervised. MoSA achieves SOTA performance among unsupervised methods only using a ResNet-50 backbone and delivers results comparable to SAM [ 15 ] .

Methods	Backbone	% of SA-1B	Point	Point
Methods	(# params)	% of SA-1B	(Max)	(Oracle)
SAM [ 15 ] (supervised)	ViT-B/8 (85M)	100%	52.1	68.2
UnSAM [ 38 ]	Swin-Tiny (25M)	1%	40.3	59.5
MoSA ( ours )	Swin-Tiny (25M)	1%	41.6	63.4

Table 3 : Quantitative comparison on the point-based promptable image segmentation task. MoSA achieves superior performance over the previous SOTA, UnSAM [ 38 ] , which is not strictly unsupervised as it employs CascadePSP [ 7 ] .

Method	AR ${}_{\text{1000}}$	AR ${}_{\text{s}}$	AR ${}_{\text{m}}$	AR ${}_{\text{l}}$
BCE+Dice	3.8	1.2	2.8	6.6
Slot Attention [ 14 ]	8.2	5.2	8.6	8.7
PGCL (ours)	28.3	14.7	26.6	36.1

Table 4 : Ablation study of PGM training.

ROSM	MFSM	AR ${}_{\text{1000}}$	AR ${}_{\text{s}}$	AR ${}_{\text{m}}$	AR ${}_{\text{l}}$
✓		28.8	20.6	31.1	28.7
	✓	34.7	13.7	34.9	43.1
✓	✓	36.6	20.6	35.4	45.1

Table 5 : Ablation study of modules in the MGMS pipeline.

Figure 5 : Impact of the number of prediction heads (left) and pre-training video data scale (right) in PGM.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Examples from our synthetic video dataset, showcasing challenging scenarios with significant occlusions and complex object interactions. Each row presents one example. Columns 1-5: Five sequential video frames. Column 6: Optical flow for the central frame. Column 7: Individual instance masks for each moving object in the central frame. Column 8: The complete motion foreground segmentation mask for the central frame.

Config	ROSM	MFSM
Optimizer	AdamW [ 25 ]	AdamW [ 25 ]
Base learning rate	$5\times 10^{-4}$	$1\times 10^{-4}$
Weight decay	$1\times 10^{-4}$	$1\times 10^{-4}$
Optimizer momentum	$\beta_{1}=0.9,\beta_{2}=0.999$	$\beta_{1}=0.9,\beta_{2}=0.999$
Batch size	32	32
Learning rate schedule	Linear Decay	Linear Decay

Appendix

Table A1 : Training Hyperparameters for ROSM and MFSM.

Hyperparameter	Value
Optimizer	AdamW [ 25 ]
Base Learning Rate	1e-4
Weight Decay	1e-2
Optimizer Momentum	$\beta_{1}=0.9,\beta_{2}=0.999$
Batch Size	32
Learning Rate Schedule	Linear Decay

Appendix

Table A2: Training Hyperparameters for the Perceptual Grouping Model (PGM).

	PtIn	LVIS	Entity	PACO	ADE	COCO	SA-1B	Average
Only maskness	28.6	24.3	24.5	13.9	21.2	27.6	28.9	24.1
Only sharpness	26.3	22.7	23.5	12.8	20.0	27.1	27.2	22.8
Maskness + sharpness	34.7	27.9	28.2	16.1	25.5	32.6	36.6	28.8

Appendix

Table A3 : Ablation of mask-quality filtering. We report AR on seven evaluation benchmarks and their average.

	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$ (Mean)	$\mathcal{F}$ (Mean)
OCLR [ 40 ]	55.1	54.5	55.7
VideoCutLER [ 37 ]	57.3	57.4	57.2
MGMS (ours)	55.9	55.8	56.0

Appendix

Table A4 : Zero-shot video segmentation on DAVIS2017.

Figure A2 : Scaling trend with video pre-training data. Compared with PGM trained using VideoCutLER pseudo-labels, PGM trained with our motion-driven pseudo-labels scales more robustly as video data increases.

Figure A3 : Rigid Object Segmentation Module (ROSM) output examples. For each case, the first three columns show key input frames, and the fourth column presents ROSM’s segmentation of the dominant rigidly moving entity. ROSM effectively segments whole rigid objects (cars, spoon), isolates articulated parts (elephant’s ear/trunk, human head), and distinguishes smaller moving components (leaves). Furthermore, ROSM can leverage parallax cues from camera motion to segment static objects(e.g., table next to the dog).

Figure A4 : Motion Foreground Segmentation Module (MFSM) effectively extracts all moving foreground elements. For each case, the first three columns display representative input frames, while the fourth column shows MFSM’s segmentation results.

Figure A5 : Additional qualitative results for whole-image segmentation, comparing our MoSA (rightmost column) with the supervised SAM [ 15 ] (second column) and the previous state-of-the-art unsupervised method UnSAM [ 38 ] (third column). Note that UnSAM is not strictly unsupervised, as its refinement stage utilizes the supervised CascadePSP [ 7 ] refiner.

Figure A6 : Comparative results for promptable image segmentation, evaluating our unsupervised MoSA against supervised SAM [ 15 ] and unsupervised UnSAM [ 38 ] across diverse scenario. The examples illustrate that when prompted by points (indicated as star marks), MoSA consistently generates multi-granular segmentation masks of higher visual quality. Note that UnSAM is not strictly unsupervised, as its refinement stage utilizes the supervised CascadePSP [ 7 ] refiner.

Explore similar work

Jul 8, 2026cs.CV

`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

Video object segmentation (VOS) is a fundamental task in video understanding, requiring accurate delineation and consistent tracking of objects across frames. While supervised methods achieve strong performance, they rely on densely annotated datasets that are costly to obtain and have limited domain coverage. Self-supervised learning offers a promising alternative by removing the need for manual labels; however, existing approaches often struggle to jointly maintain spatial accuracy and temporal coherence, particularly in unconstrained multi-object scenarios. Many rely on optical flow, synthetic motion cues, or task-specific pretraining, limiting scalability and generalisation. We propose a self-supervised framework, Cross-Temporal Consistency and Clustering, that learns mid-level, part-aware representations by combining attention-guided token selection with lightweight temporal clustering. Instead of operating at the pixel or whole-object level, the method aligns soft part assignments across time using a saliency-weighted symmetric consistency objective. The framework leverages a frozen transformer backbone with lightweight modules for adaptive token selection and multi-offset temporal alignment, enabling efficient scaling across resolutions and motion patterns.

Waqas Arshid, Mohammad Awrangjeb, Alan Wee-Chung Liew +1

School of Information and Communication Technology, Griffith University, Brisbane, QLD, Australia. · School of Engineering and Built Environment – Electrical and Electronic Engineering, Griffith University, Brisbane, QLD, Australia.

Video Object Segmentation Self-Supervised Learning Optical Flow Objects Clustering

Ask PDF

Jun 3, 2026cs.CV

Scene-Centric Unsupervised Video Panoptic Segmentation

Video panoptic segmentation (VPS) aims to jointly detect, segment, and track all objects while partitioning the video into semantically consistent regions. We introduce the task setting of unsupervised VPS, omitting any human supervision. Existing unsupervised scene understanding works mainly focused on image segmentation tasks; the video domain remains underexplored. We propose VideoCUPS, the first unsupervised VPS approach. VideoCUPS generates temporally consistent panoptic video pseudo-labels from scene-centric videos by exploiting unsupervised depth, motion, and visual cues. Training on these pseudo-labels using a novel Video DropLoss yields an accurate, unsupervised VPS model. To benchmark progress, we introduce a comprehensive evaluation protocol and four competitive baselines, extending state-of-the-art unsupervised panoptic image and instance video segmentation models to VPS. VideoCUPS outperforms all baselines and demonstrates strong label-efficient learning. With VideoCUPS, our evaluation protocol, and baselines, we provide a strong foundation for future research on unsupervised VPS.

Christoph Reich, Oliver Hahn, Nikita Araslanov +4

TU Munich · TU Darmstadt · MCML +3

Video Object Segmentation Panoptic Segmentation

Ask PDF

Apr 27, 2026cs.CV

X2SAM: Any Segmentation in Images and Videos

Multimodal Large Language Models (MLLMs) have demonstrated strong image-level visual understanding and reasoning, yet their pixel-level perception across both images and videos remains limited. Foundation segmentation models such as the SAM series produce high-quality masks, but they rely on low-level visual prompts and cannot natively interpret complex conversational instructions. Existing segmentation MLLMs narrow this gap, but are usually specialized for either images or videos and rarely support both textual and visual prompts in one interface. We introduce X2SAM, a unified segmentation MLLM that extends any-segmentation capabilities from images to videos. Given conversational instructions and visual prompts, X2SAM couples an LLM with a Mask Memory module that stores guided vision features for temporally consistent video mask generation. The same formulation supports generic, open-vocabulary, referring, reasoning, grounded conversation generation, interactive, and visual grounded segmentation across image and video inputs. We further introduce the Video Visual Grounded (V-VGD) segmentation benchmark, which evaluates whether a model can segment object tracks in videos from interactive visual prompts. With a unified joint training strategy over heterogeneous image and video datasets, X2SAM delivers strong video segmentation performance, remains competitive on image segmentation benchmarks, and preserves general image and video chat ability.

Hao Wang, Limeng Qiao, Chi Zhang +4

Sun Yat-Sen University · 2 Peng Cheng Laboratory · 3 Meituan Inc

Video Object Segmentation Multimodal Large Language Models

Ask PDF