Accurately anticipating how complex, diverse scenes will evolve requires models that represent uncertainty, simulate along extended interaction chains, and efficiently explore many plausible futures. Yet most existing approaches rely on dense video or latent-space prediction, expending substantial capacity on dense appearance rather than on the underlying sparse trajectories of points in the scene. This makes large-scale exploration of future hypotheses costly and limits performance when long-horizon, multi-modal motion is essential. We address this by formulating the prediction of open-set future scene dynamics as step-wise inference over sparse point trajectories. Our autoregressive diffusion model advances these trajectories through short, locally predictable transitions, explicitly modeling the growth of uncertainty over time. This dynamics-centric representation enables fast rollout of thousands of diverse futures from a single image, optionally guided by initial constraints on motion, while maintaining physical plausibility and long-range coherence. We further introduce OWM, a benchmark for open-set motion prediction based on diverse in-the-wild videos, to evaluate accuracy and variability of predicted trajectory distributions under real-world uncertainty. Our method matches or surpasses dense simulators in predictive accuracy while achieving orders-of-magnitude higher sampling speed, making open-set future prediction both scalable and practical. Project page: http://compvis.github.io/myriad.
Figures & tables
Figure 2: Motion Token Construction. The fourier-embedded motion Δxt(i) (alternatively a zero-vector) is combined with a per-trajectory unique randomized trajectory identifier idtraj(i) and the local image features, retrieved at the current xt(i) and original position xt=0(i) , providing information about what it is and local context.
Figure 3: Positional Encoding Scheme. We encode the current and original spatial position of each token, alongside its time . Motion tokens attend to each other and to image tokens.
Figure 4: Fast Reasoning Blocks. (a) Previous methods [ 9 , cf.] use normal transformer layers, incurring significant overhead due to the multitude of operations performed per block. (b) Our fused layers reduce complexity significantly, improving efficiency.
Figure 5: Our attention mask.
Figure 6: Posterior FM Head. Left: Our FM Head consists of multiple FFN blocks conditioned on zt(i) and flow matching time τ via adaptive norms [ 47 ] . We set up the conditioning mechanism such that every component can be cached, reducing computations. Right: Our multiscale, tanh-saturated input stack helps stabilize behavior when modeling motion with heavy-tailed behavior.
Figure 7: Value distribution.
Figure 8: OWM Composition. We curate OWM to cover a wide variety of settings. Top: dataset statistics. Bottom: some examples.
Figure 9: (a) Given different input pokes (initial motion), our model produces different motions (visualized as green lines) that adhere to constraints of the environment. (b) Our model predicts coherent motion for multiple points on the same object.
Table 9
Table 2: Planning Billiard Shots through Future Exploration. Left: We compare the Accuracy of landing a ball at a randomly selected goal position in a billiard simulation by unrolling potential futures starting from varying cue ball impulses. Under a fixed compute budget, our model surpasses dense world models from scratch using the same data. This is enabled by our methods’ low Latency , enabling us to sample a large number of potential futures. Right: We visualize results w.r.t. final target error and show its evolution over planning time for our model and an I2V baseline.
Figure 11: Planning a Billiard Shot. We search for a plan to move the red ball to the goal (top left). Our model derives a plan (top right) by predicting motion for different initial actions. Executing the action moves the ball to the desired location (bottom).
Figure 12: Posterior Uncertainty vs. Error. Starting around pixel-level ( 5121 ), our model’s posterior uncertainty is well-correlated ( green line) with true error.
Table 3: Posterior Parametrization Ablation. Substituting previously used GMM-based heads with flow matching heads leads to significant improvements in accuracy and increases convergence substantially. Adding our scale cascade improves accuracy further.
Figure A : Qualitative comparison on OWM: Our model produces motion samples that are qualitatively on par with much larger models such as WAN2.2 and MAGI-1.
Figure B : Qualitative comparison on Physics-IQ: While video generation models fail to capture the complexity of object interactions and predict simplified or no motion, our approach captures realistic physical interactions.
Figure C : Qualitative comparison on Physion: For simplified rigid body settings in Physion, video generation models hallucinate overly complex motion, while our approach is able to capture physical dynamics.
Figure D : Context informed samples: The above samples show our model’s ability to take appearance information into account when predicting motion. In both comparisons, images were sampled with the same initial poke. Images where edited using nano banana for high similarity in appearance.
Figure E : Hinted Samples Our model is capable of producing complex, coherent, and appearance-informed motion given an initial motion hint.
Figure F : Un-hinted Samples Given only appearance conditioning, our approach is able to produce physically correct and coherent motion, also show ing more complex understanding, such as that cars at an intersection should not move when pedestrians are blocking their path.
Figure G : Partially-hinted Samples Given only a single poke conditioning per example, our model produces coherent motion for queries on the same or linked objects. The Jenga example highlights that our model is able to capture multi-modality if two outcomes are possible given the same initial motion hint.
Figure H : Qualitative samples on our billiard simulation. The respective top row shows our model’s prediction given an initial impulse for the ball marked in red, where we visualize the predicted trajectory state using a frame-wise renderer; the lower row shows an overlay of the ground truth simulation with the prediction to enable comparisons. Our model can successfully predict the observed motion up to minor stochastic details.
Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at https://trajloom.github.io/.
Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu +4
McMaster University · University of British Columbia · Vector Institute +2
Robotic world models predict action-conditioned future scenes, providing a foundation for understanding action outcomes. However, turning alternative predictions into useful learning signals remains challenging: similar candidates limit informative quality comparisons, while diverging trajectories require persistent maintenance of their individual histories. We introduce FutureWorlds, a framework that unifies candidate construction, history maintenance, and learning from relative quality. Built on a multimodal discrete autoregressive model, FutureWorlds uses diverse beam search during reinforcement learning to construct candidate futures that balance confidence and diversity. Candidate-specific bounded memory preserves scene states and ensures that generation and policy scoring use matching histories. We further propose MemSPO (Memory-Conditioned Search-Guided Policy Optimization), which converts video trajectory rewards into group-relative advantages to optimize the world model. On RT-1, BridgeV2, and RoboCasa, FutureWorlds reduces LPIPS for 32-frame predictions by 14.78%, 20.84%, and 9.12%, respectively, relative to the strongest baseline on each dataset. Under fixed evaluation configurations, only 200 MemSPO updates further improve generation quality and support continued prediction beyond the training horizon. Memory ablations, decoding sensitivity analysis, and optical-flow evaluation show that these gains extend beyond visual quality to more accurate motion prediction and more consistent object states. Project page and code: https://github.com/Alexander-wu/FutureWorlds.
Predicting how a scene may evolve from partial observations requires reasoning about multiple possible futures rather than committing to a single trajectory. Existing approaches either generate appearance-dominated video predictions or sample a small number of trajectories without explicitly modeling the distribution of possible motion. We introduce Goal-Aware Representations of Future kInEmatic Latent Distributions (GARFIELD), a probabilistic model of scene kinematics that learns a structured spatio-temporal latent representation of the distribution over possible futures given an image and optional spatio-temporally sparse constraints. The same latent representation enables both joint sampling of all trajectories and direct access to the underlying motion distribution through an efficient deterministic density decoder. As a result, uncertainty about future motion can be localized to specific scene elements and timesteps and progressively refined through additional constraints. Experiments demonstrate strong motion planning performance competitive with large video generation models while sampling trajectories 97× faster. Our method further estimates motion densities two orders of magnitude faster than Monte-Carlo sampling from motion generation models, enabling interactive exploration and uncertainty-aware planning.