Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning

WonJun Moon; Hyun Seok Seong; Jae-Pil Heo

cs.CVMar 24, 2026

Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning

Authors: WonJun Moon, Hyun Seok Seong, Jae-Pil Heo

Organizations: Sungkyunkwan University, South Korea

Abstract

Video Object-Centric Learning seeks to decompose raw videos into a small set of object slots, but existing slot-attention models often suffer from severe over-fragmentation. This is because the model is implicitly encouraged to occupy all slots to minimize the reconstruction objective, thereby representing a single object with multiple redundant slots. We tackle this limitation with a reconstruction-guided slot curriculum (SlotCurri). Training starts with only a few coarse slots and progressively allocates new slots where reconstruction error remains high, thus expanding capacity only where it is needed and preventing fragmentation from the outset. Yet, during slot expansion, meaningful sub-parts can emerge only if coarse-level semantics are already well separated; however, with a small initial slot budget and an MSE objective, semantic boundaries remain blurry. Therefore, we augment MSE with a structure-aware loss that preserves local contrast and edge information to encourage each slot to sharpen its semantic boundaries. Lastly, we propose a cyclic inference that rolls slots forward and then backward through the frame sequence, producing temporally consistent object representations even in the earliest frames. All combined, SlotCurri addresses object over-fragmentation by allocating representational capacity where reconstruction fails, further enhanced by structural cues and cyclic inference. Notable FG-ARI gains of +6.8 on YouTube-VIS and +8.3 on MOVi-C validate the effectiveness of SlotCurri. Our code is available at github.com/wjun0830/SlotCurri.

Figures & tables

Figure 1: Motivation and method overview. (a) Original video frames. (b) Ground-truth masks. (c) Our baseline SlotContrast [ 4 ] learns to decompose with a high slot budget from scratch. This approach often results in over-fragmentation, splitting a single object (car) across multiple slots (brown and purple slots). (d) In contrast, our curriculum learning significantly reduces this over-fragmentation, yielding semantically coherent slots. (e) This is achieved by starting with a minimal slot budget ( e.g. , 2 slots) and progressively spawning new slots in regions that existing slots fail to capture, thereby preserving object-level slots.

Figure 2: An architectural overview of our framework.

Figure 3: Illustration of our reconstruction‑guided slot curriculum learning. The model begins with a few coarse slots that coarsely segment large regions. At each scheduled iteration, slots whose reconstruction errors ( $\delta$ ) remain high are duplicated and perturbed with distance‑aware noise ( Eq. 6 ), creating new slots that focus on the under‑explained areas. Iterating this process across training stages gradually enlarges the slot set, yielding a fine‑grained partition with clearly separated object regions.

Figure 4: Slot attention maps at an early training (Curriculum stage 1 with 2 slots). (a) Without structural loss, slots broadly capture coarse semantic divisions but exhibit blurred boundaries and partial fragmentation of object parts (helmet of person on the left), indicating weak structural consistency. (b) With structural loss, slots form clearer, sharper boundaries and more coherent object representations, significantly improving object grouping quality.

Figure 5: Illustration of cyclic inference. Slots on blue background are the output set preserving contextual information.

Method	YouTube‑VIS
Method	FG‑ARI $\uparrow$	mBO $\uparrow$
STEVE [ 24 ]	15.0	19.1
VideoSAUR [ 27 ]	28.9	26.3
VideoSAURv2 [ 27 ]	31.2	29.7
SlotContrast [ 4 ]	38.0	33.7
SlotCurri ( Ours )	44.8 ±1.2	35.5 ±2.2

Table 1: Results on real-world YouTube-VIS dataset.

Method	MOVi‑C		MOVi‑E
Method	FG‑ARI $\uparrow$	mBO $\uparrow$	FG‑ARI $\uparrow$	mBO $\uparrow$
SAVi [ 25 ]	22.2	13.6	42.8	16.0
STEVE [ 24 ]	36.1	26.5	50.6	26.6
VideoSAUR [ 27 ]	64.8 ±1.2	38.9 ±0.6	73.9 ±1.1	35.6 ±0.5
VideoSAURv2 [ 27 ]	–	–	77.1	34.4
SlotContrast [ 4 ]	69.3	32.7	82.9	29.2

Table 2: Results on synthetic MOVi-C and MOVi-E datasets.

Method	C	E
Method	Image FG‑ARI
AdaSlot [ 5 ]	75.6	76.7
SOLV [ 26 ]	-	80.8
SlotCurri	81.6	84.9

Table 3: Image FG-ARI compared against baselines targeting over-fragmentation on MOVi-C and MOVi-E datasets.

Curriculum		Struct.	Cycle	YouTube‑VIS
Simple	R-G	Struct.	Cycle	FG‑ARI $\uparrow$	mBO $\uparrow$
-	-	-	-	36.1	32.7
✓	-	-	-	38.8	32.3
✓	-	-	✓	40.3	32.3
-	✓	-	-	42.6	33.7
-	✓	✓	-	43.6	35.2

Table 4: Ablation study on YouTube-VIS. Our baseline is SlotContrast, reproduced using official code. R-G denotes reconstruction-guided slot curriculum, Struct is structure-aware reconstruction loss, and Cycle indicates the cyclic inference.

Table 5: Hyperparameter sensitivity analysis on YouTube-VIS. Bolded parameters denote default configurations.

Figure 6: Qualitative results on YouTube-VIS dataset. We present visualizations of video frames along with their corresponding GT masks and the predicted masks from SlotContrast and our method. The examples in the fourth to sixth rows (highlighted with a blue background) illustrate how our slot representations evolve across different stages of the curriculum.

Table 6: Analysis of object discovery quality against SlotContrast (SC).

Method	@0.3	@0.5	@0.7
SContrast	1.23	1.21	1.19
Ours	1.22	1.18	1.15

Table A1: Average number of GT objects per slot (Degree of under-fragmentation). A GT object is matched to a slot if

\text{IoU}\geq\rho

. Higher values indicate that single slots contain multiple objects (severe under-fragmentation).

Figure A1 : Qualitative analysis of the impact of each component.

Slot Scheduling	YouTube‑VIS
Slot Scheduling	FG‑ARI $\uparrow$	mBO $\uparrow$
Decelerated	38.2	32.9
Linear	43.4	34.2
Accelerated	44.8	35.5

Table A2: Ablation study on the scheduling strategies on YouTube-VIS.

Slot Splitting Criteria	YouTube‑VIS
Slot Splitting Criteria	FG‑ARI $\uparrow$	mBO $\uparrow$
Area Norm	40.4	32.7
Total Error	44.8	35.5

Table A3: Analysis of slot spawning criteria on YouTube-VIS. We compare our proposed criterion (Total Error Mass) against a scale-invariant alternative (Area-Normalized Error).

Figure A2 : Qualitative Comparison of Spawning Criteria.

Cycle	YouTube‑VIS
#Frames ( $C$ )	FG‑ARI $\uparrow$	mBO $\uparrow$
✗	43.6	35.2
2	43.8	35.2
3	43.9	35.1
full	44.8	35.5

Table A4: Performance of chunk-wise short cyclic inference on YouTube-VIS. We compare the full cycle, no cycle (✗), and short cycles constrained to each video chunk of

C

frames (

C\in\{2,3\}

) on YouTube-VIS.

Figure A3 : Qualitative results on MOVi-E dataset. We present visualizations of video frames along with their corresponding GT masks and prediction masks from our SlotCurri. The white circles in the first row highlight the limitation of our work.

Method	$\mathcal{L}_{\text{SSIM3D}}$	YouTube‑VIS
Method	$\mathcal{L}_{\text{SSIM3D}}$	FG‑ARI $\uparrow$	mBO $\uparrow$
SlotContrast ∗	-	36.1	32.7
SlotContrast	✓	36.4	32.7
R-G Curriculum	-	42.9	33.8
R-G Curriculum	✓	44.8	35.5

Table A5: Effectiveness of structure-aware reconstruction loss (

\mathcal{L}_{\text{SSIM3D}}

\ast

represents the reproduced results.

SSIM Objective	YouTube‑VIS
SSIM Objective	FG‑ARI $\uparrow$	mBO $\uparrow$
✗ SSIM	42.9	33.8
+2D-SSIM	44.2	34.3
+3D-SSIM	44.8	35.5

Table A6: Comparison between spatio-temporal (3D-SSIM) and spatial-only (2D-SSIM) structure-aware loss on YouTube-VIS.

Method	slot=7		slot=11		slot=15
Method	FG‑ARI $\uparrow$	mBO $\uparrow$	FG‑ARI $\uparrow$	mBO $\uparrow$	FG‑ARI $\uparrow$	mBO $\uparrow$
SContrast	74.9	27.9	69.3	32.7	61.8	31.2
Ours	78.7	27.8	77.6	32.8	74.8	33.9

Table A7: Results with varying number of slots on MOVi-C.

Method	YouTube-VIS		MOVi‑C		MOVi‑E
Method	FG‑ARI $\uparrow$	mBO $\uparrow$	FG‑ARI $\uparrow$	mBO $\uparrow$	FG‑ARI $\uparrow$	mBO $\uparrow$
Recon	27.4	28.9	50.7	25.9	70.6	24.3
SContrast	29.2	29.6	63.8	26.1	70.5	24.9
SlotCurri (Ours)	31.4	30.1	70.0	25.7	70.2	24.7

Table A8: Experimental results on object dynamics prediction. SlotFormer (SF) is used to evaluate each pretrained VOCL method.

Method	COCO
Method	Image-ARI $\uparrow$	Image-mBO $\uparrow$
Reconstruction	40.5	28.8
SlotCurri (Ours)	43.4	28.9

Table A9: Quantitative evaluation on the MS COCO dataset. SlotCurri achieves a significant improvement in Image-ARI by effectively resolving the over-fragmentation issue inherent in the reconstruction-based baseline.

Panel of Figure 25 — Figure A4 : Qualitative results on the YouTube-VIS dataset.

Panel of Figure 26 — Figure A5 : Qualitative results on the MOVi-C dataset.

Video Object-Centric Learning

See all 14 papers

Semantic Slots for Video Object-Centric Learning
Aug 21, 2026Khalil Sabri, Guillaume-Alexandre Bilodeau, Nicolas Saunier +1Video Object-Centric Learning Object-Centric Representation Learning
Polytechnique Montréal Montréal, Canada · Université TÉLUQ Montréal, Canada
Selective Synergistic Learning for Video Object-Centric Learning
Jun 14, 2026WonJun Moon, Jae-Pil HeoObject-Centric Representation Learning Video Object-Centric Learning
KAIST, South Korea · † Work done at Sungkyunkwan University. · Sungkyunkwan University, South Korea
Internalizing Temporal Consistency in Video Object-Centric Learning without Explicit Regularization
May 29, 2026Rongzhen Zhao, Zhiyuan Li, Juho Kannala +1Disentangled Representation Learning Object-Centric Representation Learning
Department of Electrical Engineering and Automation, Aalto University, Finland · Department of Computer Science, Aalto University, Finland · Center for Machine Vision and Signal Analysis, University of Oulu, Finland

Aug 21, 2026J/K move · Enter open · S save

Semantic Slots for Video Object-Centric Learning

Khalil Sabri, Guillaume-Alexandre Bilodeau, Nicolas Saunier, Wassim Bouachir

Polytechnique Montréal Montréal, Canada · Université TÉLUQ Montréal, Canada

Video Object-Centric Learning Object-Centric Representation Learning Video Understanding Video Object Segmentation

Open paper AskPDF arXiv

Video Object-Centric Learning (OCL) has traditionally focused on refining the encoder architecture to ensure temporal consistency. In this paper, we argue that the primary bottleneck lies in the decoder. We show that traditional decoders force slots to be spatially anchored, hindering their ability to adapt to motion. We propose SemanticSlots, which uses a Transformer-based decoder that leverages image context, relieving slots from encoding boundary precision and spatial location. This allows slots to function as semantic queries that are inherently object position invariant, retrieving matching features rather than memorizing coordinates. More importantly, this property allows slots computed from a single frame to decompose subsequent video frames, eliminating the need for complex temporal predictors or auxiliary temporal losses. Results on YouTube-VIS show that SemanticSlots improves upon VideoSAUR by 31 points in mBO and outperforms current state-of-the-art methods by 21 points, achieving 86.6% ARI and 62.8% mBO.

Abstract

Video Object-Centric Learning

Explore similar work

Semantic Slots for Video Object-Centric Learning

Selective Synergistic Learning for Video Object-Centric Learning

Internalizing Temporal Consistency in Video Object-Centric Learning without Explicit Regularization