Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Efstathios Karypidis; Spyros Gidaris; Nikos Komodakis

cs.CVApr 13, 2026

Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Authors: Efstathios Karypidis, Spyros Gidaris, Nikos Komodakis

Organizations: Archimedes, Athena Research Center, Greece · National Technical University of Athens · University of Crete · IACM-Forth

Abstract

Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix, a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency compared to strong diffusion baselines. We provide the implementation code at https://github.com/Sta8is/Re2Pix

Figures & tables

Figure 1: Overview of the proposed Re2Pix hierarchical framework during inference. In Stage 1, semantic features $h_{1:M}$ of the context frames are extracted from a vision foundation model (VFM) encoder $E_{h}$ and fed into a masked feature transformer to autoregressively predict (in frame-wise way) future semantic VFM features $\hat{h}_{M+1:K}$ . In Stage 2, both the past $\hat{h}_{1:M}$ and predicted features $\hat{h}_{M+1:K}$ condition the diffusion transformer $G_{z}$ to generate future VAE latents $\hat{z}_{M+1:K}$ , which are then decoded into RGB future frames.

Figure 2: Re2Pix architecture. The model takes as input (1) VAE latents $z_{1:K}$ with noise applied to the future frames $z_{M+1:K}$ , and (2) semantic features $h_{1:K}$ from a vision foundation model (VFM), with nested dropout applied to zero-out fine-grained channels. The two modalities are embedded independently and combined via channel-wise summation at the input, implementing early fusion. The diffusion transformer is trained with a denoising objective applied to the future VAE latents. Nested dropout and mixed supervision (using for the future frames both ground-truth and predicted features with a 90%/10% mixture; not shown in the figure for simplicity) regularize the model against overfitting to idealized VFM features for the future frames.

Method	Semantic Consistency				Generation
Method	mIoU(A) $\uparrow$	IoU(M) $\uparrow$	$\delta_{1}$ $\uparrow$	AbsR $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Re2Pix (Stage-1)	69.76	69.66	88.03	.1262	-	-
Baseline	60.55	57.64	85.15	.1460	12.86	60.70
w/ REPA [ 45 ]	61.45 (+0.90)	59.63 (+1.99)	85.35 (+0.20)	.1465 (+0.0005)	12.34 (-0.52)	55.15 (-5.55)
w/ VideoREPA [ 91 ]	60.98 (+0.43)	58.61 (+0.97)	85.22 (+0.07)	.1466 (+0.0006)	12.95 (+0.09)	59.03 (-1.67)
Baseline-Large	61.69 (+1.14)	59.65 (+2.01)	85.43 (+0.28)	.1453 (-0.0070)	11.99 (-0.87)	56.81 (-3.89)

Table 1: Comparison with baselines. All numbers report absolute performance; values in parentheses indicate improvement over the Baseline. Blue indicates improvement, red indicates degradation. Our hierarchical approach provides consistent gains across both semantic consistency and generation quality. Model parameters: Baseline (782M), REPA variants (792M), Baseline-Large (1.5B), Re2Pix (1.1B).

Panel of Figure 3 — Figure 3: Accelerated Training Convergence. Training curves for (a) FID, (b) FVD, and (c) Segmentation mIoU comparing our hierarchical approach (orange) with the baseline (blue). Our method achieves $\times$ 7 speed-up for generation metrics and $\times$ 14 speed-up for segmentation.

VFM forecasting only
Method	Cityscapes						nuScenes				KITTI
	Semantic Consistency				Generation		Sem. Cons.		Generation		Generation
	mIoU(A) $\uparrow$	IoU(M) $\uparrow$	$\delta_{1}$ $\uparrow$	AbsR $\downarrow$	FID $\downarrow$	FVD $\downarrow$	$\delta_{1}$ $\uparrow$	AbsR $\downarrow$	FID $\downarrow$	FVD $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Re2Pix (Stage 1)	70.46	70.26	88.03	.1208	-	-	83.82	.1869	-	-	-	-
Trained from scratch
Baseline	62.77	60.86	85.62	.1433	9.64	52.12	79.92	.2438	21.08	136.71	20.02	61.02

Table 2: Results for Re2Pix trained on extended data (Cityscapes + nuScenes + CoVLA). We report in-domain results on Cityscapes and nuScenes, zero-shot generalization on KITTI, and comparisons against Re2Pix (Stage 1), Baselines and large-scale internet-pretrained systems (Vista, Cosmos-Predict-2), both fine-tuned in our setting. For reference, we include Re2Pix (Stage 1), which reports semantic consistency metrics only, as it operates purely in feature space without the ability of generating pixels.

Table 6

Table 7

Method	Semantic Consistency				Generation
Method	mIoU(A) $\uparrow$	IoU(M) $\uparrow$	$\delta_{1}$ $\uparrow$	AbsR $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Baseline (1 run)	60.55	57.64	85.15	.1460	12.86	60.70
Re2Pix (1 run)	63.53	62.29	85.72	.1413	9.90	52.66
Baseline (3 runs)	60.92 $\pm$ 0.34	58.63 $\pm$ 0.86	85.25 $\pm$ 0.14	.1458 $\pm$ .0002	12.75 $\pm$ 0.10	63.39 $\pm$ 2.63
Re2Pix (3 runs)	63.70 $\pm$ 0.23	62.76 $\pm$ 0.51	85.75 $\pm$ 0.10	.1414 $\pm$ .0006	9.91 $\pm$ 0.05	55.16 $\pm$ 2.19

Table 7: Multiple sampling results. Single-run performance (top) and mean

\pm

standard deviation over 3 independent sampling runs (bottom) for both methods.

Table 9

Panel of Figure 4 — Figure 4: Qualitative example on Cityscapes (scene 60). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.

Panel of Figure 5 — Figure 5: Qualitative example on Cityscapes (scene 228). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.

Panel of Figure 6 — Figure 6: Qualitative example on Cityscapes (scene 289). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.

Panel of Figure 7 — Figure 7: Qualitative example on Cityscapes (scene 456). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.

Panel of Figure 8 — Figure 8: Qualitative example on Cityscapes (scene 485). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.

Video Prediction

Video prediction forecasts future frames or scene states from observed video for applications in perception, planning, and simulation.

See all 69 papers

JEPA Guided Diffusion: Predictive Vision-Language Conditioning for Generative Traffic Forecasting
Sep 18, 2026Trinh Tra Giang Nguyen, Thanh Nguyen Vo, Nguyen Hoai Thuong Bui +1Video Prediction Traffic Forecasting
Ho Chi Minh City University of Technology and Engineering, Ho Chi Minh, Viet Nam
HMPDM: A Diffusion Model for Driving Video Prediction with Historical Motion Priors
Mar 28, 2026Ke Li, Tianjia Yang, Kaidi Liang +2Video Prediction Video Diffusion Models
Department of Civil Engineering, Stony Brook University, Stony Brook, NY 11794, USA. · Department of Civil and Environmental Engineering, Pennylvania State University, University Park, PA 16802, USA.
GeoRoute: Geometry-Aware Hybrid Inference for Traffic Future-Frame Prediction
Aug 10, 2026Khang Minh Le, Hieu Dinh Trung Pham, Luu Thanh Danh +8Video Diffusion Models Autonomous Driving
PAMI Lab, Vietnamese-German University, Vietnam · University of Science, Ho Chi Minh City, Vietnam · GenAI4E Lab +2

Sep 18, 2026J/K move · Enter open · S save

JEPA Guided Diffusion: Predictive Vision-Language Conditioning for Generative Traffic Forecasting

Trinh Tra Giang Nguyen, Thanh Nguyen Vo, Nguyen Hoai Thuong Bui, Ha Duc Bui

Ho Chi Minh City University of Technology and Engineering, Ho Chi Minh, Viet Nam

Video Prediction Traffic Forecasting Video Diffusion Models

Open paper AskPDF arXiv

Accurate traffic forecasting requires both understanding scene dynamics and synthesizing realistic future observations. Recent diffusion-based video generation models produce visually plausible predictions but require expensive end-to-end training and often entangle scene understanding with image synthesis. In this work, we propose a decoupled forecasting framework that separates future representation learning from video generation. A frozen V-JEPA encoder first extracts predictive latent representations from the observed traffic videos, capturing the underlying scene dynamics in a semantic latent space. A lightweight latent alignment module then projects these representations into the conditioning space of a frozen Cosmos diffusion module, enabling future video synthesis without retraining the large generative model. By freezing all foundation models and training only the lightweight alignment module, the proposed framework substantially reduces optimization complexity while preserving forecasting capability. Experimental results on the AI City Challenge 2026 Track 5 benchmark demonstrate that the proposed method achieved a score of 75.1297, ranking third in the competition. These results suggest that predictive world representations learned by V-JEPA can effectively guide downstream video generation, providing a practical and efficient alternative to end-to-end diffusion-based forecasting.

Abstract

Video Prediction

Explore similar work

JEPA Guided Diffusion: Predictive Vision-Language Conditioning for Generative Traffic Forecasting

HMPDM: A Diffusion Model for Driving Video Prediction with Historical Motion Priors

GeoRoute: Geometry-Aware Hybrid Inference for Traffic Future-Frame Prediction