Forecasting Motion in the Wild

Neerja Thakkar; Shiry Ginosar; Jacob Walker; Jitendra Malik; Joao Carreira; Carl Doersch

cs.CVApr 1, 2026

Forecasting Motion in the Wild

Authors: Neerja Thakkar, Shiry Ginosar, Jacob Walker, Jitendra Malik, Joao Carreira, Carl Doersch

Organizations: UC Berkeley · Google DeepMind · Toyota Technological Institute at Chicago

Abstract

Visual intelligence requires anticipating the future behavior of agents, yet vision systems lack a general representation for motion and behavior. We propose dense point trajectories as visual tokens for behavior, a structured mid-level representation that disentangles motion from appearance and generalizes across diverse non-rigid agents, such as animals in-the-wild. Building on this abstraction, we design a diffusion transformer that models unordered sets of trajectories and explicitly reasons about occlusion, enabling coherent forecasts of complex motion patterns. To evaluate at scale, we curate 300 hours of unconstrained animal video with robust shot detection and camera-motion compensation. Experiments show that forecasting trajectory tokens achieves category-agnostic, data-efficient prediction, outperforms state-of-the-art baselines, and generalizes to rare species and morphologies, providing a foundation for predictive visual intelligence in the wild.

Figures & tables

Figure 1: Dense point trajectories act as visual tokens for behavior, enabling scalable prediction of complex motion across diverse species. Our method takes as input a single RGB image, a history of motion, and an optional high-level motion vector, and forecasts future animal motion in the form of point trajectories. For each predicted point trajectory, we translate a small circular patch of the input image along the motion trajectory and superimpose it on the input image ( no pixels are generated! ). Leftmost shows the start locations on the input frame; the rest is forecast by our model. Our method is capable of forecasting many different animal species and behaviors, even long-tail ones—the polar bear on the top right is only present in $0.31\%$ of the training data, the caribou on the bottom left in $0.025\%$ , and the alpaca on the bottom right in $0.50\%$ . See more results at https://motion-forecasting.github.io/.

Figure 2: Architecture. Given an input frame and (noisy) tracks, we construct a single token for every track, which includes a DINO feature at the start location, the motion history, and the noisy track values, both with occlusion indicators. After projection, we add a position encoding for the initial point location. Tokens are stacked and fed to a transformer (DiT) to predict clean tracks (right).

Figure 3: Our processed data before and after camera stabilization. Given a first frame (left), the middle image shows the point tracks in pixel space, where the motion of the animals and the camera (panning, zooming out) are entangled. On the right are our point tracks in camera-stabilized space. We release all of our annotations, including the camera-stabilized point tracks.

Figure 4: Animal motion follows a log normal distribution: We plot a histogram of animal displacement. Horizontal axis is a binned log displacement, while vertical axis is log frequency. We find that log-normal (purple) fits much better than both a power law (orange).

Figure 5: Samples from our model Sampling from our model with different random seeds (each row) and no displacement conditioning. The frame on the left is the input state after the motion history.

Figure 6: Prompting our model with different levels of motion. We ask the model for less motion (left), more motion (middle), or motion in an opposite direction (right). Grey represents motion history, colors are our predictions.

Figure 7: Out-of-distribution: Our model generalizes to humans and non-mammals.

Figure 8: Our Model vs. Stable Diffusion. Our approach can model the behavior of less common animals in our dataset such as hares, while conventional video models struggle with these animals.

Table 9

Table 10

Figure 9: Motion distribution in log-log. We plot a histogram of animal displacement. The vertical axis is log frequency for all three plots. The horizontal axis is binned log displacement magnitude (top), binned log x-displacement (bottom left), and binned log y-displacement (bottom right).

Selection	Method	FD (V) $\downarrow$	FD (A) $\downarrow$	Var (V)	Var (A)	FVMD $\downarrow$
High motion	GT	-	-	31.5	8.94	-
	No motion	27.1	7.51	0	0	481.99
	Constant vel	13.7	7.51	23.8	0	210.47
	WHN	25.2	3.19	1.1	3.34	280.77
	Track2Act	11.8	4.81	9.89	1.63	76.28
	Ours (uncond)	8.96	3.74	13.1	1.68	49.30

Table 5: Quantitative results on All Data , distribution level. FD values are multiplied by

10^{3}

; Variance values are multiplied by

10^{5}

; FVMD values are divided by

10^{3}

. Best results in bold , second best underlined .

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Selection	Method	ADE $\downarrow$	FDE $\downarrow$	VMD $\downarrow$	Avg PWT $\uparrow$
High motion	No motion	0.325	0.596	6.50	12.44%
	Constant vel	0.286	0.591	5.02	11.94%
	WHN	0.262	0.538	5.74	11.62%
	Track2Act	0.136	0.294	4.50	21.79%
	Ours (uncond)	0.119	0.275	4.33	26.01%
	Oracle vel	0.110	0.156	7.04	14.70%

Table 6: Quantitative evaluation on All Data , example-level metrics. Best results in bold , second best underlined . For non-learned baselines, we report single-sample metrics; for WHN, and ours we report best of

K=5

Selection	Method	FD (V) $\downarrow$	FD (A) $\downarrow$	Var (V)	Var (A)	FVMD $\downarrow$
High motion	GT	-	-	29.5	10.8	-
	No motion	16.6	5.61	0	0	335.406
	Constant vel	7.49	5.61	37.3	0	149.518
	WHN	15.2	3.27	1.37	4.11	247.56
	ATM	6.52	6.18	10	6.99	112.71
	Track2Act	6.32	5.06	8.01	0.446	104.85

Table 7: Quantitative results on Panthera Data , distribution level. FD values are multiplied by

10^{3}

; Variance values are multiplied by

10^{5}

; FVMD values are divided by

10^{3}

. Best results in bold , second best underlined .

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Selection	Method	ADE $\downarrow$	FDE $\downarrow$	VMD $\downarrow$	Avg PWT $\uparrow$
High motion	No motion	0.211	0.393	5.51	13.22%
	Constant vel	0.193	0.413	4.91	16.73%
	WHN	0.215	0.393	5.82	10.24%
	ATM	0.143	0.262	5.95	16.31%
	Track2Act	0.135	0.245	5.04	16.72%
	Ours (uncond)	0.107	0.209	4.77	20.68%

Table 8: Quantitative evaluation on Panthera , example-level metrics. Best results in bold , second best underlined . For non-learned baselines and ATM (single output), we report single-sample metrics; for WHN, Track2Act, and Ours we report best of

K=5

Trajectory Prediction

Trajectory prediction forecasts how people, vehicles, robots, and other dynamic systems will move from observed histories and context.

See all 187 papers

Envisioning the Future, One Step at a Time
Apr 10, 2026Stefan Andreas Baumann, Jannik Wiese, Tommaso Martorella +2Trajectory Prediction Probabilistic Trajectory Prediction
Munich Center for Machine Learning (MCML) · Netflix
MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction
Jun 17, 2026Jianing Zhang, Chenhao Zheng, Yajun Yang +10Trajectory Prediction
Allen Institute for AI · University of Washington · UNC-Chapel Hill
TrajLoom: Dense Future Trajectory Generation from Video
Mar 23, 2026Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu +4Trajectory Prediction Video Prediction
McMaster University · University of British Columbia · Vector Institute +2

Apr 10, 2026J/K move · Enter open · S save

Envisioning the Future, One Step at a Time

Stefan Andreas Baumann, Jannik Wiese, Tommaso Martorella, Mahdi M. Kalayeh+1

Munich Center for Machine Learning (MCML) · Netflix

Trajectory Prediction Probabilistic Trajectory Prediction Autoregressive Diffusion

Open paper AskPDF arXiv

Accurately anticipating how complex, diverse scenes will evolve requires models that represent uncertainty, simulate along extended interaction chains, and efficiently explore many plausible futures. Yet most existing approaches rely on dense video or latent-space prediction, expending substantial capacity on dense appearance rather than on the underlying sparse trajectories of points in the scene. This makes large-scale exploration of future hypotheses costly and limits performance when long-horizon, multi-modal motion is essential. We address this by formulating the prediction of open-set future scene dynamics as step-wise inference over sparse point trajectories. Our autoregressive diffusion model advances these trajectories through short, locally predictable transitions, explicitly modeling the growth of uncertainty over time. This dynamics-centric representation enables fast rollout of thousands of diverse futures from a single image, optionally guided by initial constraints on motion, while maintaining physical plausibility and long-range coherence. We further introduce OWM, a benchmark for open-set motion prediction based on diverse in-the-wild videos, to evaluate accuracy and variability of predicted trajectory distributions under real-world uncertainty. Our method matches or surpasses dense simulators in predictive accuracy while achieving orders-of-magnitude higher sampling speed, making open-set future prediction both scalable and practical. Project page: http://compvis.github.io/myriad.

Abstract

Trajectory Prediction

Explore similar work

Envisioning the Future, One Step at a Time

MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction

TrajLoom: Dense Future Trajectory Generation from Video