Can Unsupervised Segmentation Reduce Annotation Costs for Video Semantic Segmentation?

Samik Some; Vinay P. Namboodiri

cs.CVMar 29, 2026

Can Unsupervised Segmentation Reduce Annotation Costs for Video Semantic Segmentation?

Authors: Samik Some, Vinay P. Namboodiri

Organizations: IIT Kanpur Kanpur, India · University of Bath Bath, UK

Abstract

Present-day deep neural networks for video semantic segmentation require a large number of fine-grained pixel-level annotations to achieve the best possible results. Obtaining such annotations, however, is very expensive. On the other hand, raw, unannotated video frames are practically free to obtain. Similarly, coarse annotations, which do not require precise boundaries, are also much cheaper. This paper investigates approaches to reduce the annotation cost required for video segmentation datasets by utilising such resources. We show that using state-of-the-art segmentation foundation models, Segment Anything Model (SAM) and Segment Anything Model 2 (SAM 2), we can utilise both unannotated frames as well as coarse annotations to alleviate the effort required for manual annotation of video segmentation datasets by automating mask generation. Our investigation suggests that if used appropriately, we can reduce the need for annotation by a third with similar performance for video semantic segmentation. More significantly, our analysis suggests that the variety of frames in the dataset is more important than the number of frames for obtaining the best performance.

Figures & tables

Panel of Figure 1 — Figure 1. Examples of fine-grained and coarse annotations in the Cityscapes dataset. Examples of fine-grained and coarse annotations in the Cityscapes dataset.

Panel of Figure 2 — Figure 2. An example showing how well SAM 2 can predict masks for past and future frames, given a manually annotated frame. Here, the 20 th frame is manually annotated. Masks for both 10 th and 30 th frames were generated using SAM 2. We observe that SAM 2 can track and propagate the masks for objects present in the 20 th frame very well. However, it cannot, by design, tell us anything about new objects introduced in frames 10 and 30. An example showing how well SAM 2 can predict masks for past and future frames, given a manually annotated frame. Here, the 20\textsuperscript{th} frame is manually annotated. Masks for both 10\textsuperscript{th} and 30\textsuperscript{th} frames were generated using SAM 2. We observe that SAM 2 can track and propagate the masks for objects present in the 20\textsuperscript{th} frame very well. However, it cannot, by design, tell us anything about new objects introduced in frames 10 and 30.

TMANet
ID	Frames Used	mIoU	Manual %	Total samples
	10 15 20 25 30
N1	$\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$	75.65	100.00	2975
P2	$\blacksquare$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$	75.75	50.00	2976
P2s	$\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$ $\blacksquare$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$	74.83	50.00	2976
F2	$\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$	75.11	50.00	2976

Table 1. Mean Intersection over Union (mIoU) for training experiments on the Cityscapes dataset using manually annotated and SAM 2-generated labels mixed together. The Manual % column indicates what percentage of the training data was manually annotated. Filled boxes indicate frames that were used, whereas unfilled ones indicate unused frames. Green indicates manually annotated and black indicates SAM 2-generated masks.

Panel of Figure 3 — Figure 3. A couple of examples from the Cityscapes dataset demonstrating how well SAM refines some of the segmentation classes. As can be seen, masks for objects like cars, traffic signs, poles and people have been refined quite nicely and can be treated as fine-grained instead of being only a rough approximation. A couple of examples from the Cityscapes dataset demonstrating how well SAM refines some of the segmentation classes. As can be seen, masks for objects like cars, traffic signs, poles and people have been refined quite nicely and can be treated as fine-grained instead of being only a rough approximation.

ID	Frames Used	mIoU	Manual %	Total samples
	-10 -5 0 5 10
N1	$\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$	62.15	100.00	4032
F2	$\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$	60.67	50.00	4032
B3	$\blacksquare$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\square$ $\blacksquare$	61.31	33.33	4032

Table 2. Mean Intersection over Union (mIoU) for training experiments on the IDD dataset using manually annotated and SAM 2-generated labels mixed together. The Manual % column indicates what percentage of the training data was manually annotated. Filled boxes indicate frames that were used, whereas unfilled ones indicate unused frames. Green indicates manually annotated and black indicates SAM 2-generated masks.

Figure 4. Plot showing changes in mean Intersection over Union (mIoU) against increasing percentage of coarse annotations in the training data for TMANet on the Cityscapes dataset. The non-refined and refined values refer to the coarse annotations being used as-is versus being refined using SAM. Plot showing changes in mean Intersection over Union (mIoU) against increasing percentage of coarse annotations in the training data for TMANet on the Cityscapes dataset. The non-refined and refined values refer to the coarse annotations being used as-is versus being refined using SAM.

Coarse %	Non-refined		Refined
	mIoU	mAcc	mIoU	mAcc
$0$	76.95	86.19	76.95	86.19
$10$	75.96	84.68	76.89	86.12
$20$	72.52	83.00	74.28	84.03
$30$	74.73	83.63	75.89	84.88
$40$	75.00	83.88	75.03	84.39

Table 3. Mean IoU and Accuracy for increasing ratio of non-refined and refined coarse annotations with TMANet on the Cityscapes dataset. The Coarse % column indicates the percentage of coarse annotations in the training samples.

Figure 5. Plot showing changes in mean Intersection over Union (mIoU) against increasing percentage of coarse annotations in the training data for TDNet on the Cityscapes dataset. The non-refined and refined values refer to the coarse annotations being used as-is versus being refined using SAM. Plot showing changes in mean Intersection over Union (mIoU) against increasing percentage of coarse annotations in the training data for TDNet on the Cityscapes dataset. The non-refined and refined values refer to the coarse annotations being used as-is versus being refined using SAM.

Coarse %	Non-refined		Refined
	mIoU	mAcc	mIoU	mAcc
$0$	71.52	79.46	71.52	79.46
$50$	70.35	78.56	69.93	78.26
$100$	62.96	73.86	63.20	75.42

Table 4. Mean IoU and Accuracy for increasing ratio of non-refined and refined coarse annotations with TDNet on the Cityscapes dataset. The Coarse % column indicates the percentage of coarse annotations in the training samples.

Class	0% Coarse	20% Coarse		50% Coarse		80% Coarse		100% Coarse
		NR	R	NR	R	NR	R	NR	R
road	97.90	97.81	97.85	97.85	97.89	97.52	97.68	95.48	95.78
sidewalk	84.07	82.89	83.42	83.17	83.46	80.79	82.09	69.04	71.03
building	92.52	91.48	91.87	91.30	91.52	90.39	90.90	85.81	86.29
wall	62.24	55.68	54.54	52.53	49.19	50.47	53.74	37.19	37.51
fence	61.79	57.07	57.59	56.52	56.59	53.08	54.81	33.69	39.01

Table 5. Classwise Mean Intersection over Union for a few mixtures with TMANet on the Cityscapes dataset, for both non-refined and refined annotations.

Class	0% Coarse	50% Coarse		100% Coarse
		NR	R	NR	R
pole	51.40	49.31	50.45	42.22	42.30
t. light	57.97	54.38	53.82	44.44	46.89
t. sign	69.36	65.64	67.32	58.57	58.95
person	74.36	72.62	72.16	61.98	62.07
rider	53.63	51.89	50.64	40.84	42.68

Table 6. Classwise Mean Intersection over Union with TDNet on the Cityscapes dataset, for both non-refined and refined annotations.

Coarse %	Non-refined		Refined
	mIoU	mAcc	mIoU	mAcc
$0$	76.95	86.19	76.95	86.19
$25$	75.12	84.95	66.30	73.59
$50$	71.13	79.32	68.97	79.68
$75$	67.00	76.20	68.70	78.58
$100$	54.71	67.18	57.40	66.46

Table 7. Mean IoU and Accuracy for increasing fraction of non-refined and refined coarse annotations with TMANet using SAM 2.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision
Sep 30, 2026Weijian Jian, Xiaoyue Zhang, Bin Xiao +5Image Segmentation Self-Supervised Learning
AI Research · Independent Researcher · University of Ottawa +1
UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity
Nov 17, 2025Junwei Yu, Trevor Darrell, XuDong WangImage Segmentation Promptable Image Segmentation
UC Berkeley
Towards Automatic Video Annotation with ASH: Zero-Shot Open-Vocabulary Multi-Object Tracking and Segmentation
Oct 1, 2026Arash Rocky, Q. M. Jonathan WuVideo Object Segmentation Instance Segmentation
Department of Electrical and Computer Engineering, University of Windsor, Windsor, ON, N9B 3P4, Canada

Sep 30, 2026J/K move · Enter open · S save

Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

Weijian Jian, Xiaoyue Zhang, Bin Xiao, Chunyu Xie+4

AI Research · Independent Researcher · University of Ottawa+1

Image Segmentation Self-Supervised Learning Promptable Image Segmentation Unsupervised Image Segmentation

Open paper AskPDF arXiv

The Segment Anything Model (SAM) relies heavily on massive manual annotations, creating a fundamental bottleneck for model scaling. While unsupervised methods attempt to learn object concepts from motion, they typically overfit to moving entities, lacking both multi-granularity understanding and the ability to generalize to static objects. To overcome this, we introduce Motion-Grounded Segment Anything (MoSA), a highly scalable unsupervised framework that learns a transferable objectness prior from unlabeled videos. MoSA operates in three progressive stages: (1) automatically generating multi-granularity motion pseudo-labels from large-scale video data; (2) training a Perceptual Grouping Model (PGM) via contrastive learning to internalize a generalized, appearance-driven concept of objects; and (3) transferring this learned prior into a prompt-guided architecture for segment-anything-style inference on images. Extensive zero-shot evaluations across seven challenging benchmarks (e.g., COCO and ADE20K) demonstrate that MoSA significantly outperforms existing unsupervised methods. Notably, despite using zero manual annotations, MoSA achieves segmentation performance comparable to the fully supervised SAM. Our findings reveal that harnessing large-scale unlabeled motion is a feasible and highly scalable alternative to annotation-driven segment-anything pipelines.

Abstract

Semantic Segmentation

Explore similar work

Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity

Towards Automatic Video Annotation with ASH: Zero-Shot Open-Vocabulary Multi-Object Tracking and Segmentation