Accurate traffic forecasting requires both understanding scene dynamics and synthesizing realistic future observations. Recent diffusion-based video generation models produce visually plausible predictions but require expensive end-to-end training and often entangle scene understanding with image synthesis. In this work, we propose a decoupled forecasting framework that separates future representation learning from video generation. A frozen V-JEPA encoder first extracts predictive latent representations from the observed traffic videos, capturing the underlying scene dynamics in a semantic latent space. A lightweight latent alignment module then projects these representations into the conditioning space of a frozen Cosmos diffusion module, enabling future video synthesis without retraining the large generative model. By freezing all foundation models and training only the lightweight alignment module, the proposed framework substantially reduces optimization complexity while preserving forecasting capability. Experimental results on the AI City Challenge 2026 Track 5 benchmark demonstrate that the proposed method achieved a score of 75.1297, ranking third in the competition. These results suggest that predictive world representations learned by V-JEPA can effectively guide downstream video generation, providing a practical and efficient alternative to end-to-end diffusion-based forecasting.
Figures & tables
Figure 1: Inference overview. The trained predictor replaces Cosmos-Reason1 and directly supplies the cross attention context of the frozen Cosmos-Predict2.5 generator.
Figure 2: Training pipeline for learning the JEPA-guided conditioning context.
Figure 3: Example of the original and refined descriptions used for predictor training.
Table 1: Training and generation settings.
Method
Post-proc.
Final
PSNR ↑
SSIM ↑
LPIPS ↓
CLIP-S ↑
FID ↓
FVD ↓
VLM-guided Cosmos
No
72.0975
18.5038
0.5852
0.3152
0.9378
29.4280
25.1301
VLM-guided Cosmos
Yes
73.4072
19.3147
0.6325
0.2894
0.9347
30.6414
24.9342
JEPA-guided Cosmos-Predict2.5
No
73.5120
19.1776
0.6306
0.2963
0.9356
28.6052
26.3175
JEPA-guided Cosmos-Predict2.5
Yes
75.1297
19.7213
0.6504
0.2661
0.9452
26.5176
24.7875
Table 2: Controlled comparison of conditioning and post-processing variants.
Rank
Team
Final
PSNR ↑
SSIM ↑
LPIPS ↓
CLIP-S ↑
FID ↓
FVD ↓
1
Qyn
76.4866
20.1202
0.6503
0.2463
0.9498
22.4089
21.7907
2
SSUPER
76.0385
19.7271
0.6300
0.2487
0.9384
21.1641
19.4627
3
Latent Painter - UTE (Ours)
75.1297
19.7213
0.6504
0.2661
0.9452
26.5176
24.7875
4
CHTTL_A30
74.0544
18.8573
0.5970
0.2814
0.9423
23.7849
24.1105
5
VGU_ai_lab
73.2843
19.7360
0.6472
0.2942
0.9454
33.6081
29.3483
6
Monash Connected Autonomous Vehicles
72.0264
18.7460
0.6176
0.3074
0.9341
32.7695
31.8538
Table 3: Official AI City Challenge 2026 Track 5 leaderboard.
Generative traffic video forecasting aims to synthesize long-horizon, temporally coherent future videos of traffic scenes from a short observation history and textual descriptions. In this paper, we present CosmosAlign, a generative traffic video forecasting framework built upon the pretrained Cosmos3-Nano world foundation model. Our approach is motivated by the observation that successfully adapting large pretrained world models to downstream forecasting tasks depends primarily on distribution alignment rather than increased model capacity. To this end, we propose a two-stage LoRA adaptation strategy that first aligns the conditioning-mode distribution with the target forecasting task, and then aligns the training captions with the model's native structured prompting interface through an LLM-based re-captioning pipeline. During inference, we further improve prediction quality using a fully training-free procedure consisting of consensus-based medoid sample selection and motion-adaptive blending of static scene regions. CosmosAlign achieves a final score of 76.49 on the AI City Challenge 2026 Track 5 benchmark, ranking first on the final leaderboard. Our code is publicly available at https://quangminhdinh.github.io/CosmosAlign/.
Quang Minh Dinh, Tuan Kiet Doan
Simon Fraser University, Burnaby, BC, Canada · Institut Polytechnique de Paris, Palaiseau, France
Long-horizon future-frame prediction is important for autonomous driving, traffic surveillance, and intelligent transportation systems, yet remains challenging due to temporal ghosting, geometry drift, and inconsistent object motion. Recent latent video diffusion models have achieved impressive visual quality, but directly applying them to structured traffic scenes often leads to unstable geometry and degraded temporal coherence over extended horizons. We present a training-free inference framework that stabilizes reliable static structure in pretrained video predictions through multi-frame temporal context and view-conditioned routing. For front-camera videos, our method refines generated futures with a multi-frame depth-layered renderer that projects static geometry from observed history frames while preserving dynamic regions from the generative base model. For heterogeneous traffic views, a frozen vision-language model infers a coarse camera group from the observed clip and selects a specialized motion-based predictor. The framework requires neither retraining nor fine-tuning of the underlying video model and can be applied directly to pretrained generators. We validate the proposed framework on the AI City Challenge Track 5 benchmark, where our final system achieves competitive performance among the top-ranked teams. These results demonstrate that geometry-aware inference-time refinement and view-conditioned hybrid inference can improve static-geometry stability and low-level structural fidelity without changing the original model architecture.
Khang Minh Le, Hieu Dinh Trung Pham, Luu Thanh Danh +8
PAMI Lab, Vietnamese-German University, Vietnam · University of Science, Ho Chi Minh City, Vietnam · GenAI4E Lab +2
Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix, a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency compared to strong diffusion baselines. We provide the implementation code at https://github.com/Sta8is/Re2Pix