Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix, a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency compared to strong diffusion baselines. We provide the implementation code at https://github.com/Sta8is/Re2Pix
Figures & tables
Figure 1: Overview of the proposed Re2Pix hierarchical framework during inference. In Stage 1, semantic features h1:M of the context frames are extracted from a vision foundation model (VFM) encoder Eh and fed into a masked feature transformer to autoregressively predict (in frame-wise way) future semantic VFM features h^M+1:K . In Stage 2, both the past h^1:M and predicted features h^M+1:K condition the diffusion transformer Gz to generate future VAE latents z^M+1:K , which are then decoded into RGB future frames.
Figure 2: Re2Pix architecture. The model takes as input (1) VAE latents z1:K with noise applied to the future frames zM+1:K , and (2) semantic features h1:K from a vision foundation model (VFM), with nested dropout applied to zero-out fine-grained channels. The two modalities are embedded independently and combined via channel-wise summation at the input, implementing early fusion. The diffusion transformer is trained with a denoising objective applied to the future VAE latents. Nested dropout and mixed supervision (using for the future frames both ground-truth and predicted features with a 90%/10% mixture; not shown in the figure for simplicity) regularize the model against overfitting to idealized VFM features for the future frames.
Method
Semantic Consistency
Generation
mIoU(A) ↑
IoU(M) ↑
δ1↑
AbsR ↓
FID ↓
FVD ↓
Re2Pix (Stage-1)
69.76
69.66
88.03
.1262
-
-
Baseline
60.55
57.64
85.15
.1460
12.86
60.70
w/ REPA [ 45 ]
61.45 (+0.90)
59.63 (+1.99)
85.35 (+0.20)
.1465 (+0.0005)
12.34 (-0.52)
55.15 (-5.55)
w/ VideoREPA [ 91 ]
60.98 (+0.43)
58.61 (+0.97)
85.22 (+0.07)
.1466 (+0.0006)
12.95 (+0.09)
59.03 (-1.67)
Baseline-Large
61.69 (+1.14)
59.65 (+2.01)
85.43 (+0.28)
.1453 (-0.0070)
11.99 (-0.87)
56.81 (-3.89)
Table 1: Comparison with baselines. All numbers report absolute performance; values in parentheses indicate improvement over the Baseline. Blue indicates improvement, red indicates degradation. Our hierarchical approach provides consistent gains across both semantic consistency and generation quality. Model parameters: Baseline (782M), REPA variants (792M), Baseline-Large (1.5B), Re2Pix (1.1B).
Figure 3: Accelerated Training Convergence. Training curves for (a) FID, (b) FVD, and (c) Segmentation mIoU comparing our hierarchical approach (orange) with the baseline (blue). Our method achieves × 7 speed-up for generation metrics and × 14 speed-up for segmentation.
Method
Cityscapes
nuScenes
KITTI
Semantic Consistency
Generation
Sem. Cons.
Generation
Generation
mIoU(A) ↑
IoU(M) ↑
δ1↑
AbsR ↓
FID ↓
FVD ↓
δ1↑
AbsR ↓
FID ↓
FVD ↓
FID ↓
FVD ↓
VFM forecasting only
Re2Pix (Stage 1)
70.46
70.26
88.03
.1208
-
-
83.82
.1869
-
-
-
-
Trained from scratch
Baseline
62.77
60.86
85.62
.1433
9.64
52.12
79.92
.2438
21.08
136.71
20.02
61.02
Table 2: Results for Re2Pix trained on extended data (Cityscapes + nuScenes + CoVLA). We report in-domain results on Cityscapes and nuScenes, zero-shot generalization on KITTI, and comparisons against Re2Pix (Stage 1), Baselines and large-scale internet-pretrained systems (Vista, Cosmos-Predict-2), both fine-tuned in our setting. For reference, we include Re2Pix (Stage 1), which reports semantic consistency metrics only, as it operates purely in feature space without the ability of generating pixels.
Table 6Table 7
Method
Semantic Consistency
Generation
mIoU(A) ↑
IoU(M) ↑
δ1↑
AbsR ↓
FID ↓
FVD ↓
Baseline (1 run)
60.55
57.64
85.15
.1460
12.86
60.70
Re2Pix (1 run)
63.53
62.29
85.72
.1413
9.90
52.66
Baseline (3 runs)
60.92 ± 0.34
58.63 ± 0.86
85.25 ± 0.14
.1458 ± .0002
12.75 ± 0.10
63.39 ± 2.63
Re2Pix (3 runs)
63.70 ± 0.23
62.76 ± 0.51
85.75 ± 0.10
.1414 ± .0006
9.91 ± 0.05
55.16 ± 2.19
Table 7: Multiple sampling results. Single-run performance (top) and mean ± standard deviation over 3 independent sampling runs (bottom) for both methods.
Table 9
Figure 4: Qualitative example on Cityscapes (scene 60). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.
Figure 5: Qualitative example on Cityscapes (scene 228). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.
Figure 6: Qualitative example on Cityscapes (scene 289). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.
Figure 7: Qualitative example on Cityscapes (scene 456). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.
Figure 8: Qualitative example on Cityscapes (scene 485). Top: context frames with corresponding DINOv2 features (PCA visualization). Bottom: ground-truth future frames, predictions from the baseline, and predictions from Re2Pix with their forecasted semantic features. Red boxes highlight regions where Re2Pix provides noticeably more accurate than the baseline.
Accurate traffic forecasting requires both understanding scene dynamics and synthesizing realistic future observations. Recent diffusion-based video generation models produce visually plausible predictions but require expensive end-to-end training and often entangle scene understanding with image synthesis. In this work, we propose a decoupled forecasting framework that separates future representation learning from video generation. A frozen V-JEPA encoder first extracts predictive latent representations from the observed traffic videos, capturing the underlying scene dynamics in a semantic latent space. A lightweight latent alignment module then projects these representations into the conditioning space of a frozen Cosmos diffusion module, enabling future video synthesis without retraining the large generative model. By freezing all foundation models and training only the lightweight alignment module, the proposed framework substantially reduces optimization complexity while preserving forecasting capability. Experimental results on the AI City Challenge 2026 Track 5 benchmark demonstrate that the proposed method achieved a score of 75.1297, ranking third in the competition. These results suggest that predictive world representations learned by V-JEPA can effectively guide downstream video generation, providing a practical and efficient alternative to end-to-end diffusion-based forecasting.
Video prediction is a useful function for autonomous driving, enabling intelligent vehicles to reliably anticipate how driving scenes will evolve and thereby supporting reasoning and safer planning. However, existing models are constrained by multi-stage training pipelines and remain insufficient in modeling the diverse motion patterns in real driving scenes, leading to degraded temporal consistency and visual quality. To address these challenges, this paper introduces the historical motion priors-informed diffusion model (HMPDM), a video prediction model that leverages historical motion priors to enhance motion understanding and temporal coherence. The proposed deep learning system introduces three key designs: (i) a Temporal-aware Latent Conditioning (TaLC) module for implicit historical motion injection; (ii) a Motion-aware Pyramid Encoder (MaPE) for multi-scale motion representation; (iii) a Self-Conditioning (SC) strategy for stable iterative denoising. Extensive experiments on the Cityscapes and KITTI benchmarks demonstrate that HMPDM outperforms state-of-the-art video prediction methods with efficiency, achieving a 28.2% improvement in FVD on Cityscapes under the same monocular RGB input configuration setting. The implementation codes are publicly available at https://github.com/KELISBU/HMPDM.
Ke Li, Tianjia Yang, Kaidi Liang +2
Department of Civil Engineering, Stony Brook University, Stony Brook, NY 11794, USA. · Department of Civil and Environmental Engineering, Pennylvania State University, University Park, PA 16802, USA.
Long-horizon future-frame prediction is important for autonomous driving, traffic surveillance, and intelligent transportation systems, yet remains challenging due to temporal ghosting, geometry drift, and inconsistent object motion. Recent latent video diffusion models have achieved impressive visual quality, but directly applying them to structured traffic scenes often leads to unstable geometry and degraded temporal coherence over extended horizons. We present a training-free inference framework that stabilizes reliable static structure in pretrained video predictions through multi-frame temporal context and view-conditioned routing. For front-camera videos, our method refines generated futures with a multi-frame depth-layered renderer that projects static geometry from observed history frames while preserving dynamic regions from the generative base model. For heterogeneous traffic views, a frozen vision-language model infers a coarse camera group from the observed clip and selects a specialized motion-based predictor. The framework requires neither retraining nor fine-tuning of the underlying video model and can be applied directly to pretrained generators. We validate the proposed framework on the AI City Challenge Track 5 benchmark, where our final system achieves competitive performance among the top-ranked teams. These results demonstrate that geometry-aware inference-time refinement and view-conditioned hybrid inference can improve static-geometry stability and low-level structural fidelity without changing the original model architecture.
Khang Minh Le, Hieu Dinh Trung Pham, Luu Thanh Danh +8
PAMI Lab, Vietnamese-German University, Vietnam · University of Science, Ho Chi Minh City, Vietnam · GenAI4E Lab +2