SWT: Self-Supervised Video Object Segmentation via Sliding, Wavelet and Transportation

Zhengtong Zhu; Jiaqing Fan; Hanwen Qian; Fanzhang Li

cs.CVSep 22, 2026

SWT: Self-Supervised Video Object Segmentation via Sliding, Wavelet and Transportation

Authors: Zhengtong Zhu, Jiaqing Fan, Hanwen Qian, Fanzhang Li

Organizations: School of Computer Science and Technology, Soochow University, China

Abstract

Video Object Segmentation (VOS) aims to accurately segment target objects from consecutive video frames and track the changes of the objects in each frame of the video. Conventional VOS methods typically demand substantial quantities of pixel-level labeled video sequences for fully supervised learning, which limits the performance of the model in sparse video scenes, while existing VOS methods have limited adaptability to global changes in objects. Based on this observation, in this paper, we propose self-supervised VOS with Sliding window, Wavelet transform and optimal Transport (SWT), a self-supervised VOS framework entirely trained on static dataset using contrastive learning. Firstly, a rolling sample buffer reuses overlapping groups of independently sampled images across successive updates. Secondly, to address the long-distance modeling difficulty caused by simple convolutional structures, we introduce wavelet transform to expand the receptive field of convolutional kernels, thus improving the model's representational capability. Finally, we incorporate optimal transport to assist the model in finding the globally optimal match between the target across two frames, improving the model's ability to handle nonrigid deformations of objects. SWT only requires training on the COCO dataset once and achieves excellent results on five VOS datasets as well as an additional body part propagation dataset. The code will be released soon at https://github.com/machine928/SWT.git.

Figures & tables

Figure 1. Comparison with current self-supervised VOS approaches. Based on the contrastive learning paradigm, existing methods (a) typically rely on unlabeled video data for training, while our method (b) only requires randomly cropped image pairs from static images as training samples. Compared to existing methods, our Wavelet Transform (WT) encoder achieves a larger effective receptive field, and we employ optimal transport mechanism to learn cross-frame object correspondence, which improves our method a stronger long-distance modeling capability. The yellow box in (b) represents that our method is able to form a better cross-frame correspondence of objects.

Figure 2. The pipeline of our proposed method. Training-time: Independently sampled static-image mini-batches are organized by the rolling buffer described in Section 3.2 . Two images $I_{1},I_{2}\in\mathbb{R}^{3\times h\times w}$ are then cropped on the static image with a random aspect ratio of 3/4 to 4/3, resize them, and use them as model inputs. These two images are used to compute a similarity matrix $A\in\mathbb{R}^{HW\times HW}$ , where elements with similarity below a set threshold are set to 0 and others to 1. $I_{1}$ and $I_{2}$ are fed into query encoder and key encoder respectively to obtain features $q,k\in\mathbb{R}^{3\times H\times W}$ . After grouping and fusing $q$ and $k$ , we perform cross correlation to generate marginal distributions, then use optimal transport to minimize the transport cost as spatial matching $L_{spatial}$ . Next, $q$ and $k$ are concatenated in different orders and fed into a dynamic projector to generate two pseudo-dynamic representations $P_{1},P_{2}\in\mathbb{R}^{2\times HW}$ , which are used to compute temporal matching $L_{temporal}$ . $L_{spatial}$ and $L_{temporal}$ are each multiplied by matrix $A$ to produce spatial loss $\mathcal{L}_{spatial}$ and temporal loss $\mathcal{L}_{temporal}$ , which are then summed as the final loss $\mathcal{L}_{hybrid}$ . Inference-time: Given the initial frame’s mask $M_{0}$ , the initial frame is encoded by query encoder and stored in memory bank along with $M_{0}$ . For subsequent query frames, after encoding, masked attention is performed between the query features and memory features to retrieve the top K most similar memory masks. These masks are then weighted and fused to predict the query frame’s mask, which is finally stored in memory.

Figure 3. The pipeline of wavelet convolution. The architecture employs a cascaded wavelet transform framework. For the input feature map, we first apply a 2D wavelet transform to decompose the image into four channels: low-frequency, horizontal, vertical, and diagonal high-frequency. Convolution operations are then performed on these four channels in the wavelet domain. The results are reconstructed via inverse wavelet transform and used as residuals, which are combined with the original convolved input through residual connections to produce the final output. Notably, during the second-level wavelet transform, we only process the low-frequency channel.

Item	Setting
Static training data	COCO train split; two random crops per image
Input augmentation	Random resized crop; area scale $[0.0,1.0]$ ; aspect ratio $[3/4,4/3]$ ; resized to $256\times 256$
Normalization	Mean $(0.4914,0.4822,0.4465)$ ; std. $(0.2023,0.1994,0.2010)$
Backbone and projection	ResNet-18; strides removed from layer3/layer4; projection hidden/output dimension 256; prediction hidden dimension 256
Key encoder update	Momentum coefficient initialized as 0.99 and increased to 1 with a cosine schedule
Optimization	Adam; learning rate $10^{-3}$ ; weight decay 0; 20 epochs; seed 42; mixed precision enabled

Table 1. Implementation hyperparameters used for the reported SWT model.

Method	BackBone	Training Dataset	DAVIS17-val
Method	BackBone	Training Dataset	$\mathcal{J}_{m}$	$\mathcal{J}_{r}$	$\mathcal{F}_{m}$	$\mathcal{F}_{r}$	$\mathcal{J}\&\mathcal{F}_{m}$
MuG ( Lu et al., 2020 )	ResNet-18	OxUvA	52.6	57.4	56.1	58.1	54.3
MAST ( Lai et al., 2020 )	ResNet-18	YouTubeVOS	63.3	73.2	67.6	77.7	65.5
CRW ( Jabri et al., 2020 )	ResNet-18	Kinetics	64.8	76.1	70.2	82.1	67.6
ContrastCorr ( Wang et al., 2021a )	ResNet-18	TrackingNet	60.5	-	65.5	-	63.0
VFS ( Xu and Wang, 2021 )	ResNet-18	Kinetics	64.0	-	69.4	-	66.7

Table 2. Quantitative comparison to methods on DAVIS17-val. The three highest-scores are highlighted in colored text: red , blue and green .

Method	Supervised Mode	DAVIS17-test
Method	Supervised Mode	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
CRW ( Jabri et al., 2020 )	Self-supervised	52.3	59.6	55.9
VFS ( Xu and Wang, 2021 )	Self-supervised	53.1	61.6	57.3
SCC ( Son, 2022 )	Self-supervised	55.9	64.0	59.9
SWT(Ours)	Self-supervised	57.4	66.1	61.8

Table 3. Quantitative comparison to methods on DAVIS17-test. The three highest-scores are highlighted in colored text: red , blue and green .

Method	Supervised Mode	DAVIS16-val
Method	Supervised Mode	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
SiamMask ( Wang et al., 2019a )	Semi-supervised	71.7	67.8	70.0
OSVOS ( Caelles et al., 2017 )	Semi-supervised	79.8	80.6	80.2
Colorization ( Vondrick et al., 2018 )	Self-supervised	38.9	30.0	34.9
CorrFlow ( Lai and Xie, 2019 )	Self-supervised	47.1	49.9	48.0
TimeCycle ( Wang et al., 2019b )	Self-supervised	55.8	51.1	53.5

Table 4. Quantitative comparison to methods on DAVIS16-val. The three highest-scores are highlighted in colored text: red , blue and green .

Method	Supervised Mode	Mean	Seen		Unseen
Method	Supervised Mode	Mean	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
OSVOS ( Caelles et al., 2017 )	Semi-supervised	58.8	59.8	60.5	54.2	60.7
PreMVOS ( Luiten et al., 2018 )	Semi-supervised	66.9	71.4	75.9	56.5	63.7
TVOS ( Zhang et al., 2020 )	Semi-supervised	67.8	67.1	69.4	63.0	71.6
Colorization ( Vondrick et al., 2018 )	Self-supervised	38.9	43.1	38.6	36.6	37.4
CorrFlow ( Lai and Xie, 2019 )	Self-supervised	46.6	50.6	46.6	43.8	45.7

Table 5. Quantitative comparison to methods on YoutubeVOS18-val set. The three highest-scores are highlighted in colored text: red , blue and green .

Method	Supervised Mode	Mean	Seen		Unseen
Method	Supervised Mode	Mean	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
Colorization ( Vondrick et al., 2018 )	Self-supervised	39.0	43.3	38.2	36.6	37.5
CorrFlow ( Lai and Xie, 2019 )	Self-supervised	47.0	51.2	46.6	44.5	45.9
RHMNet ( Zhou et al., 2025 )	Scribble-supervised	59.3	63.9	64.8	51.7	56.7
MAST ( Lai et al., 2020 )	Self-supervised	64.9	64.3	65.3	61.5	68.4
SWT(Ours)	Self-supervised	71.7	69.5	67.7	71.6	77.8

Table 6. Quantitative comparison to methods on YoutubeVOS19-val set. The three highest-scores are highlighted in colored text: red , blue and green .

Method	Supervised Mode	VIP-val
Method	Supervised Mode	mIoU
ResNet ( He et al., 2016 )	Fully-supervised	31.9
ATEN ( Zhou et al., 2018 )	Fully-supervised	37.9
TimeCycle ( Wang et al., 2019b )	Self-supervised	28.9
UVC ( Li et al., 2019 )	Self-supervised	34.1
CLTC ( Jeon et al., 2021 )	Self-supervised	37.8

Table 7. Quantitative results for body part propagation on VIP val set. The three highest-scores are highlighted in colored text: red , blue and green .

Module Variants	S	W	T	DAVIS17-val	DAVIS16-val
Module Variants	S	W	T	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
Baseline	✗	✗	✗	72.0	78.9
S	✓	✗	✗	72.3	79.3
W	✗	✓	✗	72.6	79.6
T	✗	✗	✓	73.1	80.2
S + W	✓	✓	✗	73.0	80.3

Table 8. Ablation studies on individual modules. The three highest-scores are highlighted in colored text: red , blue and green .

Augmentation Strategy	DAVIS17-val	DAVIS16-val
Augmentation Strategy	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$
Detection Cropping	71.9	78.8
Enhanced Detection Cropping	72.8	79.9
Random Cropping	73.3	80.5
Marginal Distributions
Uniform	73.2	80.4

Table 9. Ablation study on augmentation strategies and marginal distributions. The highest score is shown in bold.

Method	Params	FLOPs	GPU memory	FPS
Baseline	0.753M	1.865G	16.0GB	48
SWT	1.160M	2.064G	21.4GB	43

Table 11. Efficiency comparison between the baseline and SWT.

Figure 4. Ablation results of the wavelet transform and optimal transport modules. The four columns from left to right correspond to: baseline, adding only the wavelet transform module, adding only the optimal transport module, and incorporating both modules, respectively. The yellow boxes circle the area of accuracy improvement.

Panel of Figure 5 — Figure 5. Further analysis of convergence under different sliding-window buffer settings and effective receptive fields under different backbone configurations.

Panel of Figure 6 — Figure 6. This is a further experimental analysis of the effectiveness of Optimal Transport.

Figure 7. Qualitative results on the DAVIS dataset. From top to bottom, the first row, second row, third row and fourth row show qualitative results for DAVIS17-val and DAVIS17-test respectively. The temporal direction of video sequences proceeds from left to right.

Figure 8. Qualitative results on the YouTubeVOS dataset. From top to bottom, the first and second rows show qualitative results for YouTubeVOS18-val, while the third and fourth rows display results for YouTubeVOS19-val. The temporal direction of video sequences proceeds from left to right.

Figure 9. Qualitative results on the VIP dataset. The temporal direction of video sequences proceeds from left to right.

Video Object Segmentation

Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.

See all 114 papers

`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation
Jul 8, 2026Waqas Arshid, Mohammad Awrangjeb, Alan Wee-Chung Liew +1Video Object Segmentation Self-Supervised Learning
School of Information and Communication Technology, Griffith University, Brisbane, QLD, Australia. · School of Engineering and Built Environment – Electrical and Electronic Engineering, Griffith University, Brisbane, QLD, Australia.
Learning Accurate Segmentation Purely from Self-Supervision
Feb 27, 2026Zuyao You, Zuxuan Wu, Yu-Gang JiangUnsupervised Image Segmentation Salient Object Detection
Fudan University
SlotMatch: Distilling Object-Centric Representations for Unsupervised Video Segmentation
Aug 5, 2025Diana-Nicoleta Grigore, Neelu Madan, Andreas Mogelmose +2Video Object Segmentation Video Object-Centric Learning
University of Bucharest, Romania · Aalborg University, Denmark

Jul 8, 2026J/K move · Enter open · S save

`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

Waqas Arshid, Mohammad Awrangjeb, Alan Wee-Chung Liew, Yongsheng Gao

School of Information and Communication Technology, Griffith University, Brisbane, QLD, Australia. · School of Engineering and Built Environment – Electrical and Electronic Engineering, Griffith University, Brisbane, QLD, Australia.

Video Object Segmentation Self-Supervised Learning Video Representation Learning Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Video object segmentation (VOS) is a fundamental task in video understanding, requiring accurate delineation and consistent tracking of objects across frames. While supervised methods achieve strong performance, they rely on densely annotated datasets that are costly to obtain and have limited domain coverage. Self-supervised learning offers a promising alternative by removing the need for manual labels; however, existing approaches often struggle to jointly maintain spatial accuracy and temporal coherence, particularly in unconstrained multi-object scenarios. Many rely on optical flow, synthetic motion cues, or task-specific pretraining, limiting scalability and generalisation. We propose a self-supervised framework, Cross-Temporal Consistency and Clustering, that learns mid-level, part-aware representations by combining attention-guided token selection with lightweight temporal clustering. Instead of operating at the pixel or whole-object level, the method aligns soft part assignments across time using a saliency-weighted symmetric consistency objective. The framework leverages a frozen transformer backbone with lightweight modules for adaptive token selection and multi-offset temporal alignment, enabling efficient scaling across resolutions and motion patterns.

Abstract

Video Object Segmentation

Explore similar work

`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

Learning Accurate Segmentation Purely from Self-Supervision

SlotMatch: Distilling Object-Centric Representations for Unsupervised Video Segmentation