Visual intelligence requires anticipating the future behavior of agents, yet vision systems lack a general representation for motion and behavior. We propose dense point trajectories as visual tokens for behavior, a structured mid-level representation that disentangles motion from appearance and generalizes across diverse non-rigid agents, such as animals in-the-wild. Building on this abstraction, we design a diffusion transformer that models unordered sets of trajectories and explicitly reasons about occlusion, enabling coherent forecasts of complex motion patterns. To evaluate at scale, we curate 300 hours of unconstrained animal video with robust shot detection and camera-motion compensation. Experiments show that forecasting trajectory tokens achieves category-agnostic, data-efficient prediction, outperforms state-of-the-art baselines, and generalizes to rare species and morphologies, providing a foundation for predictive visual intelligence in the wild.
Figures & tables
Figure 1: Dense point trajectories act as visual tokens for behavior, enabling scalable prediction of complex motion across diverse species. Our method takes as input a single RGB image, a history of motion, and an optional high-level motion vector, and forecasts future animal motion in the form of point trajectories. For each predicted point trajectory, we translate a small circular patch of the input image along the motion trajectory and superimpose it on the input image ( no pixels are generated! ). Leftmost shows the start locations on the input frame; the rest is forecast by our model. Our method is capable of forecasting many different animal species and behaviors, even long-tail ones—the polar bear on the top right is only present in 0.31% of the training data, the caribou on the bottom left in 0.025% , and the alpaca on the bottom right in 0.50% . See more results at https://motion-forecasting.github.io/.
Figure 2: Architecture. Given an input frame and (noisy) tracks, we construct a single token for every track, which includes a DINO feature at the start location, the motion history, and the noisy track values, both with occlusion indicators. After projection, we add a position encoding for the initial point location. Tokens are stacked and fed to a transformer (DiT) to predict clean tracks (right).
Figure 3: Our processed data before and after camera stabilization. Given a first frame (left), the middle image shows the point tracks in pixel space, where the motion of the animals and the camera (panning, zooming out) are entangled. On the right are our point tracks in camera-stabilized space. We release all of our annotations, including the camera-stabilized point tracks.
Figure 4: Animal motion follows a log normal distribution: We plot a histogram of animal displacement. Horizontal axis is a binned log displacement, while vertical axis is log frequency. We find that log-normal (purple) fits much better than both a power law (orange).
Figure 5: Samples from our model Sampling from our model with different random seeds (each row) and no displacement conditioning. The frame on the left is the input state after the motion history.
Figure 6: Prompting our model with different levels of motion. We ask the model for less motion (left), more motion (middle), or motion in an opposite direction (right). Grey represents motion history, colors are our predictions.
Figure 7: Out-of-distribution: Our model generalizes to humans and non-mammals.
Figure 8: Our Model vs. Stable Diffusion. Our approach can model the behavior of less common animals in our dataset such as hares, while conventional video models struggle with these animals.
Table 9Table 10
Figure 9: Motion distribution in log-log. We plot a histogram of animal displacement. The vertical axis is log frequency for all three plots. The horizontal axis is binned log displacement magnitude (top), binned log x-displacement (bottom left), and binned log y-displacement (bottom right).
Selection
Method
FD (V) ↓
FD (A) ↓
Var (V)
Var (A)
FVMD ↓
High motion
GT
-
-
31.5
8.94
-
No motion
27.1
7.51
0
0
481.99
Constant vel
13.7
7.51
23.8
0
210.47
WHN
25.2
3.19
1.1
3.34
280.77
Track2Act
11.8
4.81
9.89
1.63
76.28
Ours (uncond)
8.96
3.74
13.1
1.68
49.30
Table 5: Quantitative results on All Data , distribution level. FD values are multiplied by 103 ; Variance values are multiplied by 105 ; FVMD values are divided by 103 . Best results in bold , second best underlined . ↑ indicates higher is better; ↓ indicates lower is better.
Selection
Method
ADE ↓
FDE ↓
VMD ↓
Avg PWT ↑
High motion
No motion
0.325
0.596
6.50
12.44%
Constant vel
0.286
0.591
5.02
11.94%
WHN
0.262
0.538
5.74
11.62%
Track2Act
0.136
0.294
4.50
21.79%
Ours (uncond)
0.119
0.275
4.33
26.01%
Oracle vel
0.110
0.156
7.04
14.70%
Table 6: Quantitative evaluation on All Data , example-level metrics. Best results in bold , second best underlined . For non-learned baselines, we report single-sample metrics; for WHN, and ours we report best of K=5 .
Selection
Method
FD (V) ↓
FD (A) ↓
Var (V)
Var (A)
FVMD ↓
High motion
GT
-
-
29.5
10.8
-
No motion
16.6
5.61
0
0
335.406
Constant vel
7.49
5.61
37.3
0
149.518
WHN
15.2
3.27
1.37
4.11
247.56
ATM
6.52
6.18
10
6.99
112.71
Track2Act
6.32
5.06
8.01
0.446
104.85
Table 7: Quantitative results on Panthera Data , distribution level. FD values are multiplied by 103 ; Variance values are multiplied by 105 ; FVMD values are divided by 103 . Best results in bold , second best underlined . ↑ indicates higher is better; ↓ indicates lower is better.
Selection
Method
ADE ↓
FDE ↓
VMD ↓
Avg PWT ↑
High motion
No motion
0.211
0.393
5.51
13.22%
Constant vel
0.193
0.413
4.91
16.73%
WHN
0.215
0.393
5.82
10.24%
ATM
0.143
0.262
5.95
16.31%
Track2Act
0.135
0.245
5.04
16.72%
Ours (uncond)
0.107
0.209
4.77
20.68%
Table 8: Quantitative evaluation on Panthera , example-level metrics. Best results in bold , second best underlined . For non-learned baselines and ATM (single output), we report single-sample metrics; for WHN, Track2Act, and Ours we report best of K=5 .
Accurately anticipating how complex, diverse scenes will evolve requires models that represent uncertainty, simulate along extended interaction chains, and efficiently explore many plausible futures. Yet most existing approaches rely on dense video or latent-space prediction, expending substantial capacity on dense appearance rather than on the underlying sparse trajectories of points in the scene. This makes large-scale exploration of future hypotheses costly and limits performance when long-horizon, multi-modal motion is essential. We address this by formulating the prediction of open-set future scene dynamics as step-wise inference over sparse point trajectories. Our autoregressive diffusion model advances these trajectories through short, locally predictable transitions, explicitly modeling the growth of uncertainty over time. This dynamics-centric representation enables fast rollout of thousands of diverse futures from a single image, optionally guided by initial constraints on motion, while maintaining physical plausibility and long-range coherence. We further introduce OWM, a benchmark for open-set motion prediction based on diverse in-the-wild videos, to evaluate accuracy and variability of predicted trajectory distributions under real-world uncertainty. Our method matches or surpasses dense simulators in predictive accuracy while achieving orders-of-magnitude higher sampling speed, making open-set future prediction both scalable and practical. Project page: http://compvis.github.io/myriad.
Stefan Andreas Baumann, Jannik Wiese, Tommaso Martorella +2
Munich Center for Machine Learning (MCML) · Netflix
Motion forecasting is central to visual intelligence: agents must anticipate how objects will move in order to plan actions, reason about physical interactions, and synthesize realistic futures. We argue that 3D points in world coordinates provide a general representation that is class-agnostic, view-stable, compact, and directly useful for downstream tasks. We formalize the task of goal-conditioned 3D point motion forecasting: given a short visual history, a set of 3D query points on an object of interest, and a language description of the intended goal, the model predicts the future 3D trajectory of each point. We introduce a full stack to study this task at scale: (1) MolmoMotion-1M is a large corpus of action-described, object-grounded 3D point trajectories annotated from 1.16M unconstrained videos; (2) PointMotionBench is a human-verified benchmark spanning 111 object categories and 61 motion types; and (3) MolmoMotion is a general motion forecasting model that supports both autoregressive coordinate prediction and flow-matching-based trajectory generation. MolmoMotion accurately predicts diverse motion patterns with different language instructions, and significantly outperforms existing motion prediction baselines on PointMotionBench. Finally, we show that the learned 3D motion prior transfers well to downstream applications: it improves training efficiency and generalization for robot manipulation, and its predicted trajectories provide effective motion guidance for generative models to synthesize videos with more realistic object motion.
Jianing Zhang, Chenhao Zheng, Yajun Yang +10
Allen Institute for AI · University of Washington · UNC-Chapel Hill
Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at https://trajloom.github.io/.
Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu +4
McMaster University · University of British Columbia · Vector Institute +2