cs.CVSep 30, 2026
SaveSeeing as Humans Do: Learning from Motion to Segment Anything Without Supervision
Organizations: AI Research · Independent Researcher · University of Ottawa · Beijing University of Posts and Telecommunications
Abstract
The Segment Anything Model (SAM) relies heavily on massive manual annotations, creating a fundamental bottleneck for model scaling. While unsupervised methods attempt to learn object concepts from motion, they typically overfit to moving entities, lacking both multi-granularity understanding and the ability to generalize to static objects. To overcome this, we introduce Motion-Grounded Segment Anything (MoSA), a highly scalable unsupervised framework that learns a transferable objectness prior from unlabeled videos. MoSA operates in three progressive stages: (1) automatically generating multi-granularity motion pseudo-labels from large-scale video data; (2) training a Perceptual Grouping Model (PGM) via contrastive learning to internalize a generalized, appearance-driven concept of objects; and (3) transferring this learned prior into a prompt-guided architecture for segment-anything-style inference on images. Extensive zero-shot evaluations across seven challenging benchmarks (e.g., COCO and ADE20K) demonstrate that MoSA significantly outperforms existing unsupervised methods. Notably, despite using zero manual annotations, MoSA achieves segmentation performance comparable to the fully supervised SAM. Our findings reveal that harnessing large-scale unlabeled motion is a feasible and highly scalable alternative to annotation-driven segment-anything pipelines.
Figures & tables
Figure 1 : Overview of our proposed MoSA . The top-left and top-right panels respectively show qualitative comparisons of MoSA against the fully supervised SAM [ 15 ] and the unsupervised UnSAM [ 38 ] on point-based promptable segmentation and whole-image segmentation. The middle-right panel shows that MoSA achieves SOTA performance compared to other unsupervised segmentation methods. The bottom panel illustrates the pipeline of MoSA’s three-stage unsupervised framework. Note that UnSAM is not strictly unsupervised, as it utilizes the supervised CascadePSP [ 7 ] refiner.
Figure 2 : An overview of our Multi-Granularity Motion Segmentation (MGMS) pipeline for generating high-quality pseudo-labels from unlabeled videos. The pipeline first extracts bidirectional optical flow from videos and processes it through two parallel modules: the Rigid Object Segmentation Module and the Motion Foreground Segmentation Module. These generate masks for rigid parts and the moving foreground of the central frame (highlighted in red). Finally, masks are filtered and processed with NMS to yield the final multi-granularity pseudo-labels.
Figure 3 : Illustration of Perceptual Grouping Contrastive Learning (PGCL). Our PGM features multiple heads, each trained on pseudo-masks of a specific scale range. For each assigned mask, we compute a similarity map by comparing an in-mask anchor patch ( + ) with all patches in the corresponding head’s feature space. The PGCL loss supervises this map using the mask as the ground truth, training the model to group coherent patches into objects.
| Methods | PtIn | LVIS | Entity | PACO | ADE | COCO | SA-1B | Average |
|---|---|---|---|---|---|---|---|---|
| UnSAM [ 38 ] (CRF [ 18 ] ) | 23.2 | 16.5 | 22.1 | 10.8 | 15.1 | 21.9 | 15.3 | 17.8 |
| UnSAM [ 38 ] (CascadePSP † [ 7 ] ) | 27.3 | 19.9 | 24.3 | 12.8 | 17.8 | 24.8 | 24.7 | 21.7 |
| PGM ( ours ) (w/o refinement) | 34.7 | 27.9 | 28.2 | 16.1 | 25.5 | 32.6 | 36.6 | 28.8 |
| PGM ( ours ) (CRF [ 18 ] ) | 36.1 | 29.3 | 30.1 | 16.9 | 26.7 | 33.8 | 37.8 | 30.1 |
Table 1 : Evaluation results of PGM on seven datasets. † : The pseudo-labels used by UnSAM [ 38 ] are not strictly unsupervised as they are refined by CascadePSP [ 7 ] .
Figure 4 : Qualitative comparison of PGM (ours) with DINOv3 family [ 32 ] , where the red + denotes anchor patches.
| Methods | Backbone | # | Avg. | with Whole Entities | with Parts | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| #params | images | COCO | LVIS | ADE | Entity | SA-1B | PtIn | PACO | ||
| SAM [ 15 ] | VIT-B(85M) | 11M | 42.1 | 49.6 | 46.1 | 45.8 | 45.9 | 60.8 | 28.3 | 18.1 |
| FreeSOLO [ 34 ] | RN-101 (45M) | 1.3M | 7.3 | 11.6 | 5.9 | 7.3 | 8.0 | 2.2 | 13.8 | 2.4 |
| CutLER [ 36 ] | RN-50 (23M) | 1.3M | 21.8 | 28.1 | 20.2 | 26.3 | 23.1 | 17.0 | 28.7 | 8.9 |
| SOHES [ 3 ] | VIT-B(85M) | 0.2M | 30.1 | 30.5 | 29.1 | 31.1 | 33.5 | 33.3 | 36.0 | 17.1 |
| UnSAM † [ 38 ] | RN-50(23M) | 0.1M | 39.2 | 40.5 | 37.7 | 35.7 | 39.6 | 41.9 | 51.6 | 27.5 |
Table 2 : Results on the whole-image segmentation task. † : UnSAM [ 38 ] utilizes CascadePSP [ 7 ] , a method that relies on manual annotations, and is therefore not strictly unsupervised. MoSA achieves SOTA performance among unsupervised methods only using a ResNet-50 backbone and delivers results comparable to SAM [ 15 ] .
| Methods | Backbone | % of SA-1B | Point | Point |
| (# params) | (Max) | (Oracle) | ||
| SAM [ 15 ] (supervised) | ViT-B/8 (85M) | 100% | 52.1 | 68.2 |
| UnSAM [ 38 ] | Swin-Tiny (25M) | 1% | 40.3 | 59.5 |
| MoSA ( ours ) | Swin-Tiny (25M) | 1% | 41.6 | 63.4 |
Table 3 : Quantitative comparison on the point-based promptable image segmentation task. MoSA achieves superior performance over the previous SOTA, UnSAM [ 38 ] , which is not strictly unsupervised as it employs CascadePSP [ 7 ] .
| Method | AR | AR | AR | AR |
|---|---|---|---|---|
| BCE+Dice | 3.8 | 1.2 | 2.8 | 6.6 |
| Slot Attention [ 14 ] | 8.2 | 5.2 | 8.6 | 8.7 |
| PGCL (ours) | 28.3 | 14.7 | 26.6 | 36.1 |
Table 4 : Ablation study of PGM training.
| ROSM | MFSM | AR | AR | AR | AR |
|---|---|---|---|---|---|
| ✓ | 28.8 | 20.6 | 31.1 | 28.7 | |
| ✓ | 34.7 | 13.7 | 34.9 | 43.1 | |
| ✓ | ✓ | 36.6 | 20.6 | 35.4 | 45.1 |
Table 5 : Ablation study of modules in the MGMS pipeline.
Figure 5 : Impact of the number of prediction heads (left) and pre-training video data scale (right) in PGM.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Examples from our synthetic video dataset, showcasing challenging scenarios with significant occlusions and complex object interactions. Each row presents one example. Columns 1-5: Five sequential video frames. Column 6: Optical flow for the central frame. Column 7: Individual instance masks for each moving object in the central frame. Column 8: The complete motion foreground segmentation mask for the central frame.
| Config | ROSM | MFSM |
|---|---|---|
| Optimizer | AdamW [ 25 ] | AdamW [ 25 ] |
| Base learning rate | ||
| Weight decay | ||
| Optimizer momentum | ||
| Batch size | 32 | 32 |
| Learning rate schedule | Linear Decay | Linear Decay |
Table A1 : Training Hyperparameters for ROSM and MFSM.
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW [ 25 ] |
| Base Learning Rate | 1e-4 |
| Weight Decay | 1e-2 |
| Optimizer Momentum | |
| Batch Size | 32 |
| Learning Rate Schedule | Linear Decay |
Table A2: Training Hyperparameters for the Perceptual Grouping Model (PGM).
| PtIn | LVIS | Entity | PACO | ADE | COCO | SA-1B | Average | |
|---|---|---|---|---|---|---|---|---|
| Only maskness | 28.6 | 24.3 | 24.5 | 13.9 | 21.2 | 27.6 | 28.9 | 24.1 |
| Only sharpness | 26.3 | 22.7 | 23.5 | 12.8 | 20.0 | 27.1 | 27.2 | 22.8 |
| Maskness + sharpness | 34.7 | 27.9 | 28.2 | 16.1 | 25.5 | 32.6 | 36.6 | 28.8 |
Table A3 : Ablation of mask-quality filtering. We report AR on seven evaluation benchmarks and their average.
| (Mean) | (Mean) | ||
|---|---|---|---|
| OCLR [ 40 ] | 55.1 | 54.5 | 55.7 |
| VideoCutLER [ 37 ] | 57.3 | 57.4 | 57.2 |
| MGMS (ours) | 55.9 | 55.8 | 56.0 |
Table A4 : Zero-shot video segmentation on DAVIS2017.
Figure A2 : Scaling trend with video pre-training data. Compared with PGM trained using VideoCutLER pseudo-labels, PGM trained with our motion-driven pseudo-labels scales more robustly as video data increases.
Figure A3 : Rigid Object Segmentation Module (ROSM) output examples. For each case, the first three columns show key input frames, and the fourth column presents ROSM’s segmentation of the dominant rigidly moving entity. ROSM effectively segments whole rigid objects (cars, spoon), isolates articulated parts (elephant’s ear/trunk, human head), and distinguishes smaller moving components (leaves). Furthermore, ROSM can leverage parallax cues from camera motion to segment static objects(e.g., table next to the dog).
Figure A4 : Motion Foreground Segmentation Module (MFSM) effectively extracts all moving foreground elements. For each case, the first three columns display representative input frames, while the fourth column shows MFSM’s segmentation results.
Figure A5 : Additional qualitative results for whole-image segmentation, comparing our MoSA (rightmost column) with the supervised SAM [ 15 ] (second column) and the previous state-of-the-art unsupervised method UnSAM [ 38 ] (third column). Note that UnSAM is not strictly unsupervised, as its refinement stage utilizes the supervised CascadePSP [ 7 ] refiner.
Figure A6 : Comparative results for promptable image segmentation, evaluating our unsupervised MoSA against supervised SAM [ 15 ] and unsupervised UnSAM [ 38 ] across diverse scenario. The examples illustrate that when prompted by points (indicated as star marks), MoSA consistently generates multi-granular segmentation masks of higher visual quality. Note that UnSAM is not strictly unsupervised, as its refinement stage utilizes the supervised CascadePSP [ 7 ] refiner.