DiDA: Video Object Segmentation with Distillation Learning of Deformable Attention

Quang-Trung Truong; Duc Thanh Nguyen; Binh-Son Hua; Sai-Kit Yeung

cs.CVJan 25, 2024

DiDA: Video Object Segmentation with Distillation Learning of Deformable Attention

Authors: Quang-Trung Truong, Duc Thanh Nguyen, Binh-Son Hua, Sai-Kit Yeung

Organizations: Hong Kong University of Science and Technology · Deakin University · Trinity College Dublin

Abstract

Video object segmentation is a fundamental research problem in computer vision. Recent techniques have often applied attention mechanism to object representation learning from video sequences. However, due to temporal changes in the video data, attention maps may not well align with the objects of interest across video frames, causing accumulated errors in long-term video processing. In addition, existing techniques have utilised complex architectures, requiring highly computational complexity and hence limiting the ability to integrate video object segmentation into low-powered devices. To address these issues, we propose DiDA, a new method for video object segmentation based on Distillation Learning of Deformable Attention. Specifically, we devise a lightweight architecture for video object segmentation that is effectively adapted to temporal changes. This is enabled by deformable attention mechanism, where the keys and values capturing the memory of a video sequence in the attention module have flexible locations updated across frames. The learnt object representations are thus adaptive to both the spatial and temporal dimensions. We train the proposed architecture using a new knowledge distillation paradigm where deformable attention maps are integrated into the distillation loss. We qualitatively and quantitatively evaluate our method and compare it with existing methods on benchmark datasets including DAVIS 2016/2017 and YouTube-VOS 2018/2019. Experimental results verify the superiority of our method via its achieved state-of-the-art performance on YouTube-VOS18 dataset and optimal memory usage. Project page and code: https://github.com/quangtrungtruong/DiDA.

Figures & tables

Panel of Figure 1 — Figure 1 : (a) From left to right: input frame and ground-truth segmentation masks, distilled feature map and segmentation masks by the distillation strategy used in MobileVOS [ 30 ] , distilled feature map and segmentation masks by our method. (b) Segmentation accuracy ( $\mathcal{J}$ & $\mathcal{F}$ ) of MobileVOS [ 30 ] and our method with different numbers of query objects in DAVIS-17 val (see more details in Section 4 ).

Figure 2 : Summary of our proposed VOS method . (a) Overview of our knowledge distillation method. The teacher model transfers intermediate attention maps to the student model. This transfer is enforced by a CKA-based loss $\mathcal{L}_{att}$ . At the same time, probability distributions of logits are transferred using intra-object and inter-object losses $\mathcal{L}_{intra}$ and $\mathcal{L}_{inter}$ . Both the teacher and student models make use of Gated Propagation Module (GPM) [ 49 ] , aiming to propagate spatio-temporal information across frames via the attention mechanism. (b) Our proposed Gated Deformable Attention function, which is used to implement the GPM. (c) Deformable attention module, which is used to replace the vanilla attention in the Gated deformable attention function.

Attention mechanism	DAVIS-16 Val			DAVIS-17 Val			YT-VOS18	YT-VOS19
	$\mathcal{J}$ & $\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	$\mathcal{J}$ & $\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	$\mathcal{J}$ & $\mathcal{F}\uparrow$	$\mathcal{J}$ & $\mathcal{F}\uparrow$
Vanilla attention	83.35	83.30	83.40	69.10	66.60	71.60	68.61	69.26
Deformable attention	85.75	84.90	86.60	72.75	69.90	75.60	73.18	73.95

Table 1 : Comparison of the vanilla attention and deformable attention in attention learning in VOS. Best performances are highlighted.

Methods	DAVIS-16 Val			DAVIS-17 Val			YT-VOS18	YT-VOS19
	$\mathcal{J}$ & $\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	$\mathcal{J}$ & $\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	$\mathcal{J}$ & $\mathcal{F}\uparrow$	$\mathcal{J}$ & $\mathcal{F}\uparrow$
DIST [ 18 ]	84.50	84.00	85.00	71.05	68.50	73.60	71.40	72.90
PEFD [ 5 ]	-	-	-	57.20	54.80	59.60	-	-
MobileVOS [ 30 ]	80.1	79.5	80.7	70.3	67.9	72.7	72.68	72.96
Ours	85.75	84.90	86.60	72.75	69.90	75.60	73.18	73.95

Table 2 : Comparison of state-of-the-art knowledge distillation methods. Best performances are highlighted. For [ 5 ] , we re-executed the supplied code from the original work, but were not able to produce reasonable results on the Davis-16 and YT-VOS18/19 datasets. We, therefore, fill the results of [ 5 ] on those datasets with “-”.

Figure 3 : Qualitative results of our method and a baseline (a DeAOTT model with the vanilla attention trained using the standard KD, i.e., only logit layers are transferred). As shown, compared with the baseline, our method can maintain the association of the objects and their IDs across frames (see frame 36 in the 1st and 2nd row). Our method also tends to be aware of parts of the same object (see frame 50 in the 3rd and 4th row). We hypothesize this success is due to the cross-frame adaptivity of deformable attention over its counterpart. More results are provided in our supplementary material.

Methods	DAVIS-17 Val				YouTube-VOS18
	$\mathcal{J}$ & $\mathcal{F}\uparrow$	$\mathcal{J}\uparrow$	$\mathcal{F}\uparrow$	FPS $\uparrow$	$\mathcal{J}$ & $\mathcal{F}\uparrow$
CorrFlow [ 21 ]	50.30	48.40	52.20	2.00	-
MAST [ 22 ]	65.50	63.30	67.60	2.06	64.20
Self-cycle [ 40 ]	70.50	67.40	73.60	-	-
Mining [ 19 ]	70.30	67.90	72.60	-	67.30
LIIR [ 24 ]	72.10	69.70	74.50	1.87	69.30

Table 3 : Comparison of VOS methods in terms of segmentation accuracy (

\mathcal{J}\&\mathcal{F}

\mathcal{J}

\mathcal{F}

) and inference speed (frame-per-second - FPS) on DAVIS-17 Val dataset. Best and second-best performances are highlighted with bold and underlines, respectively. Works in [ 40 , 19 ] do not provide executable code for re-production. Their inference speed, thus, is not reported. Methods in [ 21 , 40 ] , on the other hand, are not evaluated on the YouTube-VOS18 dataset. Note that we use pseudo-labels to train the student network, while the remaining models are trained in a self-supervised manner.

Table 8

Explore similar work

Aug 12, 2026cs.CV

Distractor-Aware Video Object Segmentation

Semi-supervised video object segmentation is a challenging task that aims to segment a target throughout a video sequence given an initial mask at the first frame. Discriminative approaches have demonstrated competitive performance on this task at a sensible complexity. These approaches typically formulate the problem as a one-versus-one classification between the target and the background. However, in reality, a video sequence usually encompasses a target, background, and possibly other distracting objects. Those objects increase the risk of introducing false positives, especially if they share visual similarities with the target. Therefore, it is more effective to separate distractors from the background, and handle them independently. We propose a one-versus-many scheme to address this situation by separating distractors into their own class. This separation allows imposing special attention to challenging regions that are most likely to degrade the performance. We demonstrate the prominence of this formulation by modifying the learning-what-to-learn (LWL) method to be distractor-aware. Our proposed approach sets a new state-of-the-art on the DAVIS 2017 val dataset, and improves over the baseline on the DAVIS 2017 test-dev benchmark by 4.6 percentage points.

Andreas Robinson, Abdelrahman Eldesokey, Michael Felsberg

Linköping University, Sweden

Video Object Segmentation Distractor

Ask PDF

Jul 8, 2026cs.CV

`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

Video object segmentation (VOS) is a fundamental task in video understanding, requiring accurate delineation and consistent tracking of objects across frames. While supervised methods achieve strong performance, they rely on densely annotated datasets that are costly to obtain and have limited domain coverage. Self-supervised learning offers a promising alternative by removing the need for manual labels; however, existing approaches often struggle to jointly maintain spatial accuracy and temporal coherence, particularly in unconstrained multi-object scenarios. Many rely on optical flow, synthetic motion cues, or task-specific pretraining, limiting scalability and generalisation. We propose a self-supervised framework, Cross-Temporal Consistency and Clustering, that learns mid-level, part-aware representations by combining attention-guided token selection with lightweight temporal clustering. Instead of operating at the pixel or whole-object level, the method aligns soft part assignments across time using a saliency-weighted symmetric consistency objective. The framework leverages a frozen transformer backbone with lightweight modules for adaptive token selection and multi-offset temporal alignment, enabling efficient scaling across resolutions and motion patterns.

Waqas Arshid, Mohammad Awrangjeb, Alan Wee-Chung Liew +1

School of Information and Communication Technology, Griffith University, Brisbane, QLD, Australia. · School of Engineering and Built Environment – Electrical and Electronic Engineering, Griffith University, Brisbane, QLD, Australia.

Video Object Segmentation Self-Supervised Learning Optical Flow Objects Clustering

Ask PDF

Jun 10, 2026cs.CV

Dual-State Slot Attention: Decoupling Appearance and Identity for Video Object-Centric Learning

Unsupervised video object-centric learning aims to decompose dynamic scenes into persistent, object-level representations without supervision. However, existing slot-based methods struggle to maintain stable object identity in challenging settings such as rapid motion and partial occlusion. First, they typically encode both the per-frame appearance of an object and its identity across frames in a single slot vector, creating an objective conflict that leads to slot swapping: reconstruction requires sensitivity to transient visual changes, whereas temporal consistency requires invariance to them. Second, the token renormalization used in Slot Attention can amplify weakly attending slots, allowing them to absorb tokens from other objects and destabilize slot-to-object correspondence. We propose Dual-State Slot Attention (DSSA), a fully self-supervised framework that addresses these limitations by separating appearance from identity and by reducing spurious updates from weakly matching slots. DSSA decomposes each slot into a local state for per-frame appearance and an identity state for temporally stable object information, thereby aligning reconstruction and temporal consistency with separate representations. The identity state is updated through a learned recurrent transition that acts as a temporal filter on the local state, while competition-modulated aggregation (CMA) down-weights updates from weakly matching slots and prevents them from absorbing tokens from other objects. Experiments on MOVi-C, MOVi-D, and YouTube-VIS demonstrate that DSSA consistently improves segmentation quality and temporal consistency over prior methods, while also yielding stronger downstream object recognition and video dynamics prediction. Code and models will be made publicly available upon acceptance.

Sieu Tran, Duc Nguyen, Hao Vo +2

University of Arkansas Fayetteville, AR, USA

Video Object Segmentation Object-Centric Representations Spatio-Temporal Attention

Ask PDF