Semantic Slots for Video Object-Centric Learning

Khalil Sabri; Guillaume-Alexandre Bilodeau; Nicolas Saunier; Wassim Bouachir

cs.CVAug 21, 2026

Semantic Slots for Video Object-Centric Learning

Authors: Khalil Sabri, Guillaume-Alexandre Bilodeau, Nicolas Saunier, Wassim Bouachir

Organizations: Polytechnique Montréal Montréal, Canada · Université TÉLUQ Montréal, Canada

Abstract

Video Object-Centric Learning (OCL) has traditionally focused on refining the encoder architecture to ensure temporal consistency. In this paper, we argue that the primary bottleneck lies in the decoder. We show that traditional decoders force slots to be spatially anchored, hindering their ability to adapt to motion. We propose SemanticSlots, which uses a Transformer-based decoder that leverages image context, relieving slots from encoding boundary precision and spatial location. This allows slots to function as semantic queries that are inherently object position invariant, retrieving matching features rather than memorizing coordinates. More importantly, this property allows slots computed from a single frame to decompose subsequent video frames, eliminating the need for complex temporal predictors or auxiliary temporal losses. Results on YouTube-VIS show that SemanticSlots improves upon VideoSAUR by 31 points in mBO and outperforms current state-of-the-art methods by 21 points, achieving 86.6% ARI and 62.8% mBO.

Figures & tables

Figure 1: Position Invariance via Context-Aware Decoding. We test whether slots computed from one frame can decompose a spatially shifted version of that frame. Row 1: Original frame and translated version. Row 2: MLP decoder [ 15 ] . Row 3: SlotMixer decoder [ 14 ] . Row 4: SemanticSlots-Frozen (ours). Context-free decoders memorize the original positions and fail under displacement; our context-aware decoder re-localizes the object using the shifted feature map as the search space.

Figure 2: Architecture motivation and overview of SemanticSlots. (a) Standard recurrent video OCL maintains temporal consistency through recurrent slot updates, predictors, and temporal losses. (b) SemanticSlots is trained only on static images using a reconstruction loss. (c) SemanticSlots-Frozen computes slots once from the first frame and reuses them as fixed semantic queries for all later frames. (d) SemanticSlots-Refined uses the same image-trained checkpoint, but refreshes slots at each frame during inference, without predictors, temporal losses, or video-level training.

	MOVi-C (#slot=11)				MOVi-D (#slot=21)
Method	ARI	ARI fg	mBO	mIoU	ARI	ARI fg	mBO	mIoU
STEVE	–	–	–	–	32.7 $\pm$ 0.2	66.5 $\pm$ 0.2	23.0 $\pm$ 0.3	21.2 $\pm$ 0.3
VideoSAUR	41.9 $\pm$ 1.1	53.3 $\pm$ 2.1	16.1 $\pm$ 0.4	14.8 $\pm$ 0.4	–	–	–	–
SlotContrast	64.6 $\pm$ 9.4	59.9 $\pm$ 5.3	27.7 $\pm$ 3.0	25.8 $\pm$ 2.9	45.3 $\pm$ 4.1	63.9 $\pm$ 0.2	26.7 $\pm$ 1.0	25.1 $\pm$ 0.9
RandSF.Q (tsim)	64.0 $\pm$ 2.9	66.3 $\pm$ 1.7	28.4 $\pm$ 1.3	26.1 $\pm$ 1.1	41.2 $\pm$ 2.2	72.0 $\pm$ 1.1	27.1 $\pm$ 0.9	25.4 $\pm$ 0.9
RandSF.Q (ssc)	65.4 $\pm$ 10.7	67.4 $\pm$ 2.1	29.2 $\pm$ 3.8	26.8 $\pm$ 3.7	41.6 $\pm$ 3.7	77.5 $\pm$ 1.0	27.4 $\pm$ 1.0	25.6 $\pm$ 1.0

Table 1: Comparison with state-of-the-art video object discovery. Bold : best result. Underline : second best. Higher is better. Baseline results are from Zhao et al 26

Panel of Figure 3 — Figure 3: Qualitative comparison on two sequences from YouTube-VIS. Rows from top to bottom represent: (1) Input frames, (2) Ground-truth masks, (3) SlotContrast 12 , (4) RandSF.Q 26 , and (5) Our approach. Our method shows superior boundary precision and reduced classification noise compared to transition-based baselines.

Panel of Figure 4 — Figure 4: Dynamic scene recovery. Left: a new skateboarder enters at frame 4. The Frozen baseline fails by design, while the Adaptive mechanism detects the semantic shift and triggers an update, shown by the red border. Right: an object is partially occluded and blurry in the initial frames. The Adaptive mechanism does not trigger because the object is already present, whereas uniform refinement continuously improves slot quality and preserves accurate boundaries over time.

Inference Mode	Time / Video (ms)	mBO	ARI
SemanticSlots-Frozen	58.6	60.8	86.4
SemanticSlots-Adaptive	60.8	61.3	86.6
SemanticSlots-Refined	69.5	62.1	86.9

Table 2: Inference efficiency vs. segmentation quality on YTVIS.

Decoder	Context	ARI	mBO
Transformer	✓(features)	86.6	62.8
Transformer	– (pos. queries)	45.8	40.0

Table 3: Transformer decoder with and without access to frame features on YTVIS.

Method	Frames/iter	Time (h)	Relative to SemanticSlots
VideoSAUR	6	$\sim$ 22	1.4 $\times$
SlotContrast	6	$\sim$ 24	1.5 $\times$
RandSF.Q	6	$\sim$ 26	1.7 $\times$
SemanticSlots	1	$\sim$ 16	–

Table 4: Training sequence length and efficiency. Frames/iter denotes the number of frames processed per iteration. By training on single frames, SemanticSlots trains significantly faster than other methods.

Figure 5: Fixed-Slot Decomposition Stability. Comparison of mBO over 36 frames using frozen slots from the first frame. The Transformer decoder (blue), with access to features from subsequent frames, shows negligible degradation, proving it successfully retrieves object identities regardless of motion. In contrast, “blind” decoders (black, green) fail to adapt to object displacement, demonstrating the fragility of coordinate-locked representations.

Configuration	Train Data	Pred.	Tsim	Iter	mBO	ARI
VideoSAUR (baseline)	Video ( $T{=}6$ )	✓	✓	2	59.8	81.2
$-$ Tsim loss	Video ( $T{=}6$ )	✓	–	2	59.6	81.7
$-$ Iterations	Video ( $T{=}6$ )	✓	✓	0	47.8	69.9
$-$ Predictor	Video ( $T{=}6$ )	–	✓	2	43.7	49.3
Iter=1	Video ( $T{=}6$ )	✓	✓	1	42.8	54.8
SemanticSlots-Frozen	Image ( $T{=}1$ )	–	–	0	60.8	84.1

Table 5: Ablation study on YTVIS. We isolate the impact of temporal modeling components under video-level training versus our image-only training paradigm. ( ∗ Denotes iterations applied strictly at inference time).

Panel of Figure 6 — Figure 6: Comprehensive qualitative comparison across diverse visual domains. Rows within each sequence display, from top to bottom: (1) Input video frames, (2) Ground-truth instance masks, (3) SlotContrast 12 , (4) RandSF.Q 26 , and (5) Our approach (SemanticSlots). Our framework yields significantly more stable identity assignments, tracks shapes with better boundary precision, and inherently eliminates the background over-clustering noise plaguing transition-based recurrent baselines.

Video Object-Centric Learning

See all 14 papers

Selective Synergistic Learning for Video Object-Centric Learning
Jun 14, 2026WonJun Moon, Jae-Pil HeoObject-Centric Representation Learning Video Object-Centric Learning
KAIST, South Korea · † Work done at Sungkyunkwan University. · Sungkyunkwan University, South Korea
Internalizing Temporal Consistency in Video Object-Centric Learning without Explicit Regularization
May 29, 2026Rongzhen Zhao, Zhiyuan Li, Juho Kannala +1Disentangled Representation Learning Object-Centric Representation Learning
Department of Electrical Engineering and Automation, Aalto University, Finland · Department of Computer Science, Aalto University, Finland · Center for Machine Vision and Signal Analysis, University of Oulu, Finland
Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning
Mar 24, 2026WonJun Moon, Hyun Seok Seong, Jae-Pil HeoVideo Object-Centric Learning Object-Centric Representation Learning
Sungkyunkwan University, South Korea

Jun 14, 2026J/K move · Enter open · S save

Selective Synergistic Learning for Video Object-Centric Learning

WonJun Moon, Jae-Pil Heo

KAIST, South Korea · † Work done at Sungkyunkwan University. · Sungkyunkwan University, South Korea

Object-Centric Representation Learning Video Object-Centric Learning Video Representation Learning

Open paper AskPDF arXiv

Typical video object-centric learning (VOCL) approaches employ slot-based frameworks that rely on reconstruction-driven encoder-decoder architectures, where learning is mediated by two spatial maps: attention maps from the encoder and object maps from the decoder. As these two distinct maps exhibit different properties, a recent dense alignment strategy attempted to reconcile this discrepancy by enforcing agreement across all spatio-temporal patches via contrastive learning. However, this indiscriminate alignment inadvertently propagates the inherent weaknesses of each module, such as noisy encoder predictions and blurred decoder boundaries. Moreover, computing dense similarities across all pairs incurs a computational cost quadratic in the total number of spatio-temporal patches, severely limiting scalability. Motivated by this, we propose Selective Synergistic Learning (SSync). Instead of exhaustive patch-to-patch alignment, SSync prevents error propagation by selectively distilling only the most reliable cues: leveraging the encoder strictly for boundary refinement and the decoder for interior denoising. This is realized via a pseudo-labeling with linear complexity, eliminating the need for quadratic spatial comparisons. Also, to prevent the reinforcement of architectural biases like slot redundancy, we introduce a transitive pseudo-label merging that consolidates overlapping slots based on spatio-temporal activation consistency. Extensive studies demonstrate that SSync improves decomposition quality and serves as a versatile, plug-and-play module while also exhibiting exceptional robustness to slot configurations. Code is available at github.com/wjun0830/SSync.

Abstract

Video Object-Centric Learning

Explore similar work

Selective Synergistic Learning for Video Object-Centric Learning

Internalizing Temporal Consistency in Video Object-Centric Learning without Explicit Regularization

Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning