cs.LGNov 18, 2026

Reconstruction vs. Prediction Pre-Training of Discrete Video Representations for Continuous Control

Authors: Ron Kosova, Francesco Croce

Organizations: University of Helsinki · ELLIS Institute Finland, Aalto University

Abstract

Self-supervised video encoders are commonly trained to either reconstruct pixels or predict latent features, yet it remains unclear how these objectives shape representations for downstream dynamics. We present a capacity-matched study isolating this choice: two identical video encoders with a shared FSQ bottleneck are trained under autoencoding and V-JEPA objectives on a continuous control environment. We introduce a simple asymmetric discretization scheme to apply the same bottleneck to the predictive model. The resulting representations exhibit a clear trade-off: the autoencoder preserves substantially more physical state information and achieves slightly better one-step latent prediction, while the predictive encoder accumulates less error under autoregressive rollout. Structural probes suggest that this advantage is associated with greater robustness to low-level visual perturbations and stronger dependence on temporal context. These results indicate that predictive and reconstructive objectives induce distinct representations of dynamics, trading state fidelity for properties that may benefit long-horizon prediction.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction

    Jun 5, 2026Jewon Yeom, Hanseul Kim, Jeongjae Park +3Video Representation LearningPredictive Representation Learning

  2. Is the reconstruction loss culprit? An attempt to outperform JEPA

    Mar 14, 2026Alexey Potapov, Oleg Shcherbakov, Ivan KravchenkoPredictive Representation LearningAutoencoders

  3. Beyond Reconstruction Error: Analytical and Data-Driven Action Tokenization for Autoregressive Vision-Language-Action Models

    Sep 22, 2026Yuxin Yang, Gaohan He, Changxue Guan +1Vision-Language-Action ModelsVLMs for Robotics