cs.CVApr 6, 2026
SaveA Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
Organizations: Amazon · Eindhoven University of Technology · Johns Hopkins University
Abstract
Anticipating diverse future states is a central challenge in video world modeling. Discriminative world models produce a deterministic prediction that implicitly averages over possible futures, while existing generative world models remain computationally expensive. Recent work demonstrates that predicting the future in the feature space of a vision foundation model (VFM), rather than a latent space optimized for pixel reconstruction, requires significantly fewer world model parameters. However, most such approaches remain discriminative. In this work, we introduce DeltaTok, a tokenizer that encodes the VFM feature difference between consecutive frames into a single continuous "delta" token, and DeltaWorld, a generative world model operating on these tokens to efficiently generate diverse plausible futures. Delta tokens reduce video from a three-dimensional spatio-temporal representation to a one-dimensional temporal sequence, for example yielding a 1,024x token reduction with 512x512 frames. This compact representation enables tractable multi-hypothesis training, where many futures are generated in parallel and only the best is supervised. At inference, this leads to diverse predictions in a single forward pass. Experiments on dense forecasting tasks demonstrate that DeltaWorld forecasts futures that more closely align with real-world outcomes, while having over 35x fewer parameters and using 2,000x fewer FLOPs than existing generative world models. Code and weights: https://deltatok.github.io.
Figures & tables
Figure 1: Outline of DeltaWorld. Unlike large existing generative world models that require many forward passes and represent each frame with many spatial tokens, our small DeltaWorld generates multiple futures in a single forward pass by using a single delta token to encode the difference between consecutive frames.
Figure 2: Performance comparison. Compared to the generative world model Cosmos [ 1 ] , our DeltaWorld forecasts futures that better align with real-world outcomes while having over fewer parameters and using fewer FLOPs.
Figure 3: Overview of DeltaTok. Given two frames encoded by a frozen vision foundation model (VFM) into grids of patch tokens and , the DeltaTok encoder takes both as input and compresses them into a single delta token . The decoder reconstructs from and . Both encoder and decoder are Vision Transformers (ViT) [ 18 ] trained with a Mean Squared Error (MSE) loss.
Figure 4: Overview of DeltaWorld. The predictor operates entirely on delta tokens (Fig. 3 ) rather than spatial tokens, enabling efficient generation of future hypotheses. Best-of-Many training (top) backpropagates only through the best predicted delta token, so that diverse futures can be sampled in a single forward pass at inference (bottom). Shown with two context frames and two queries for illustration.
| Feature | VSPW [ 47 ] | Cityscapes [ 15 ] | KITTI [ 24 ] |
|---|---|---|---|
| Task | Segmentation | Segmentation | Depth |
| Domain | Various | Driving | Driving |
| Resolution | Various (HD) | ||
| # Sequences | 343 | 500 | 28 |
Table 1: Evaluation datasets. We evaluate segmentation and depth at short ( s) and mid ( s) prediction horizons.
| Step | GFLOPs | Time | Mem | VSPW | Cityscapes |
|---|---|---|---|---|---|
| (0) Discriminative | 959 | 44.8 | 45.4 | ||
| (1) BoM training | 12013 | 47.0 (39.4) | 46.8 (31.1) | ||
| (2) Frame compress | 6315 | 45.7 (40.3) | 42.7 (35.5) | ||
| (3) Delta compress | 6721 | 46.8 (44.4) | 48.7 (45.5) |
Table 2: Towards an efficient generative world model. Reporting mid-horizon ( s) mIoU. Steps (1-3) use during training and report best -of- during evaluation ( mean in parentheses). GFLOPs for steps (1-3) reflect generating all 20 samples, and a single prediction for step (0) . Time and Mem report training time and GPU memory relative to step (0) . Using crops.
| GFLOPs | VSPW mIoU | Cityscapes mIoU | KITTI RMSE | ||||
|---|---|---|---|---|---|---|---|
| Short | Mid | Short | Mid | Short | Mid | ||
| Copy last (lower bound) | 51.2 | 44.3 | 53.5 | 39.6 | 3.76 | 4.86 | |
| DINO-world † [ 3 ] | 54.0 | 47.9 | 62.0 | 49.8 | 3.16 | 4.07 | |
| Cosmos-4B ‡ [ 1 ] | 51.1 (49.7) | 47.0 (44.5) | 55.1 (54.9) | 49.1 (48.4) | 3.82 (3.75) | 4.08 (4.14) | |
| Cosmos-12B ‡ [ 1 ] | 51.7 (50.7) | 47.7 (45.5) | 55.3 (56.0) | 53.3 (51.2) | 3.72 (3.71) | 4.01 (4.14) | |
| DeltaWorld (Ours) | 55.4 (53.7) | 50.1 (46.7) | 65.8 (63.9) | 55.4 (51.3) | 3.00 (3.17) | 3.88 (4.17) | |
Table 3: Dense forecasting. Reporting short ( s, direct) and mid ( s, 3-step rollout) prediction horizons. Generative models report best -of- evaluation ( mean in parentheses). GFLOPs reflect generating all 20 samples for generative models and a single prediction for DINO-world. Using crops. † Our reimplementation. ‡ Both variants use another 7B diffusion decoder, dominating FLOPs.
Figure 5: Best-of-Many sample scaling. Effect of the number of training and evaluation queries on Cityscapes mid-horizon ( s) mIoU. Using crops.
Figure 6: Diverse sampled futures. Top row: four context frames and the future frame. Bottom row: four sampled DeltaWorld predictions and the oracle. In this VSPW [ 47 ] example, the pedestrian’s position and ego-camera motion lead to multiple plausible futures.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Num. samples | Duration (s) | FPS | |
|---|---|---|---|
| DINO-world [ 3 ] | 66M | 5–60 | 10–60 |
| Ours | 4M | 11 | 16 |
Table A: Training data statistics. For DINO-world [ 3 ] , we report the duration range and FPS from their paper. For ours, we report the mean duration, and all videos have the same frame rate.
| Discriminative DINO-world [ 3 ] | |
|---|---|
| Backbone (4 frames) | |
| Predictor (4-frame context) | 84.88 |
| Predictor (5-frame context) | 96.96 |
| Predictor (6-frame context) | 109.04 |
| Generative DeltaWorld (Ours) | |
| Shared once | |
Table B: GFLOPs breakdown. In DeltaWorld, the backbone and DeltaTok encoder run once, while the predictor and DeltaTok decoder are applied per generated sample. Using a three-step rollout and a four-frame context (mid-horizon), ViT-B components, and crops.
Table C: Delta tokens in the discriminative DINO-world [ 3 ] . Delta tokens also perform well within a discriminative world model. Time and Mem report per-iteration training time and GPU memory relative to the discriminative baseline. Reporting mid-horizon ( s) mIoU using crops. † Our reimplementation.
Table D: Delta tokens in the discriminative DINO-Foresight [ 35 ] . Results on Cityscapes [ 15 ] show that delta tokens transfer effectively to a different discriminative architecture, matching performance with fewer tokens. The token count indicates the total number of tokens used by the world model. Using frames. † Numbers reported in the DINO-Foresight paper [ 35 ] .
Figure A : DINO-world † [ 3 ] vs . Cosmos-12B [ 1 ] vs . DeltaWorld (Ours). Given a context of four frames, predict the fifth frame (short-horizon). Second row shows the segmentation head output on the ground-truth frames, while third, fourth, and fifth rows show the segmentation head output for the predicted future frame. In this Cityscapes example [ 15 ] , DeltaWorld provides an accurate prediction of the scene evolution. Generative models show mean features from 20 samples; DINO-world shows its single deterministic prediction. Using crops. † Our reimplementation.
Figure B : Comparing mean and best for Cosmos-12B [ 1 ] vs . DeltaWorld (Ours). Given a context of four frames, predict the seventh frame autoregressively (mid-horizon). Second column shows the depth head output on the ground-truth frames, third and fourth columns show Cosmos, and fifth and sixth columns show DeltaWorld predictions. In this KITTI example [ 24 ] , DeltaWorld’s best sample more closely matches the oracle depth layout. Using crops.
Figure C : Mid-horizon rollouts (task head visualization). Each row shows four context frames (left of the dashed line) and an autoregressive rollout from DeltaWorld (right), conditioned on random noise queries, in a single forward pass per step. Top: VSPW [ 47 ] segmentation, middle: Cityscapes [ 15 ] segmentation, bottom: KITTI [ 24 ] depth. Using crops.
Figure D : Mid-horizon rollouts (RGB visualization). Same sequences as Figure C . Context columns (left of the dashed line) show ground-truth RGB frames; future columns show the predicted features decoded into pixels. Using crops.
Figure E : Diverse mid-horizon rollouts (task head visualization). Each group of three rows shares the same four context frames (left of the dashed line) but shows three different autoregressive rollouts from DeltaWorld, each conditioned on random noise queries, in a single forward pass per step. Top: VSPW [ 47 ] segmentation, middle: Cityscapes [ 15 ] segmentation, bottom: KITTI [ 24 ] depth. Using crops.
Figure F : Diverse mid-horizon rollouts (RGB visualization). Same sequences and samples as Figure E . Context columns (left of the dashed line) show ground-truth RGB frames; future columns show the predicted features decoded into pixels. Using crops.
More in this topic
Video World Models
Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.
See all 257 papers