TrajLoom: Dense Future Trajectory Generation from Video

Zewei Zhang; Jia Jun Cheng Xian; Kaiwen Liu; Ming Liang; Hang Chu; Jun Chen; Renjie Liao

cs.CVMar 23, 2026

TrajLoom: Dense Future Trajectory Generation from Video

Authors: Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu, Ming Liang, Hang Chu, Jun Chen, Renjie Liao

Organizations: McMaster University · University of British Columbia · Vector Institute · Viggle AI · Canada CIFAR AI Chair

Abstract

Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at https://trajloom.github.io/.

Figures & tables

Figure 1: For each sequence, the model observes an 81-frame history (left) and predicts future trajectories for the next 81 frames (right). Predicted trajectories are shown at three times: early, middle, and final. Colors show the spatial order of query points.

Figure 2: Overview of our pipeline. Given observed trajectories $\mathcal{T}^{p}$ , we rasterize and encode them with Grid-Anchor Offset Encoding into a dense offset field, then compress with TrajLoom-VAE into history latents $\mathbf{z}^{p}$ . Conditioned on $\mathbf{z}^{p}$ and video features, TrajLoom-Flow generates future latents via rectified-flow integration with boundary hints, which are decoded by TrajLoom-VAE into future trajectories $\hat{\mathcal{T}}^{f}$ .

Panel of Figure 3 — Figure 3: Grid-Anchor Offset Encoding converts absolute trajectories into offset space, reducing the bias of absolute coordinates.

Figure 4: We initialize

\mathbf{z}_{0}

from scaled Gaussian noise, then add each last history token

\mathbf{z}(-1,n)

to the first future token

\mathbf{z}_{0}(0,n)

Metric	Method	Kinetics	RoboTAP	Kubric	MagicData (E)
FVMD $\downarrow$	WHN (L)	8999	7587	4872	10383
FVMD $\downarrow$	Ours	3626	2467	1338	2968
FlowTV ( $10^{2}$ ) $\downarrow$	WHN (L)	24.71	27.26	12.56	26.74
FlowTV ( $10^{2}$ ) $\downarrow$	Ours	7.53	6.03	6.03	7.11
DivCurlE ( $10^{3}$ ) $\downarrow$	WHN (L)	34.69	37.39	9.89	40.04
DivCurlE ( $10^{3}$ ) $\downarrow$	Ours	3.97	2.03	2.09	2.99

Table 1: Comparison of trajectory generation with WHN (L). FVMD and flow diagnostics (FlowTV

\times 10^{2}

, DivCurlE

\times 10^{3}

;

\downarrow

) on Kinetics, RoboTAP, Kubric (MOVi-A; same config as WHN), and MagicData (E).

Frames	Method	Kinetics	RoboTAP	Kubric	MagicData (E)
24	WHN (L)-VAE	63.09	29.35	15.97	69.88
24	Ours	1.99	1.59	0.75	1.72
81	WHN (L)-VAE	63.31	31.52	16.27	67.97
81	Ours	2.04	1.61	0.77	1.72

Table 2: Fidelity of trajectory VAE reconstruction. Visibility masked endpoint error (VEPE, (

\downarrow

)) for reconstructing 24 and 81 frame trajectory segments.

Figure 5: Comparison with WHN (L). Each row shows a dataset: Kinetics, RoboTAP, Kubric, and MagicData (E). Our method yields smoother and more coherent motion.

Metric	Method	Kinetics	RoboTAP	Kubric	MagicData (E)
FVMD-Long ( $10^{-3}$ ) $\downarrow$	w/o offset	46.61	36.43	31.66	41.97
FVMD-Long ( $10^{-3}$ ) $\downarrow$	Ours	35.06	21.75	7.24	28.27
FlowTV ( $10^{2}$ ) $\downarrow$	w/o offset	15.92	15.26	15.54	15.01
FlowTV ( $10^{2}$ ) $\downarrow$	Ours	7.65	5.96	5.03	7.11
DivCurlE ( $10^{3}$ ) $\downarrow$	w/o offset	19.10	16.27	17.66	16.13
DivCurlE ( $10^{3}$ ) $\downarrow$	Ours	4.21	1.89	1.45	3.09

Table 3: Ablation study with Grid-Anchor Offset Encoding on future 81 frames.

Figure 6: Trajectory-guided video generation with Wan-Move [ 5 ] . We use the observed history to generate a trajectory. Wan-Move then uses this trajectory, along with the condition image, to generate 81 frames, as shown in the third and fourth rows.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: 1D toy example: pointwise reconstruction loss does not guarantee realistic motion. Both smooth and jittery reconstructions of $x(t)=t$ can match $L_{\mathrm{rec}}$ , but only the smooth one aligns with ground truth motion. Consistency regularization separates realistic from unrealistic solutions.

Figure 9: Additional Wan-Move results driven by our predicted trajectories. Left: conditioning inputs (history trajectory and the conditioning image). Right: our generated 81-frame future trajectories and the corresponding 81-frame videos synthesized by Wan-Move.

Frames	Method	Kinetics	RoboTAP	Kubric	MagicData (E)
24	w/o offset	4.24	3.98	3.26	3.91
	w/o regularizer	2.99	2.35	1.42	2.65
	Ours	1.99	1.59	0.75	1.72
81	w/o offset	4.30	4.01	3.24	3.99
	w/o regularizer	3.11	2.41	1.47	3.99
	Ours	2.04	1.61	0.77	1.72

Appendix

Table 4: TrajLoom-VAE ablations. Effect of Grid-Anchor Offset Encoding and spatiotemporal consistency regularization on VEPE (

\downarrow

Figure 10: TrajLoom-VAE training curves. Masked

L_{1}

reconstruction metric vs. training step. Using Grid-Anchor Offset Encoding and the spatiotemporal regularizer yields faster, more stable convergence and a substantially lower loss than removing offsets or the regularizer.

Metric	Setting	Kinetics	RoboTAP	Kubric	MagicData (E)
FVMD-Long ( $10^{-3}$ ) $\downarrow$	w/o fusion	39.60	28.86	16.38	33.87
	w/o anchoring	43.50	33.12	20.29	37.11
	20 k steps	40.13	28.81	17.02	33.51
	w/o on-policy	38.87	27.38	13.86	31.71
	Ours	35.06	21.75	7.24	28.27
FlowTV ( $10^{2}$ ) $\downarrow$	w/o fusion	9.66	8.66	9.56	8.50

Appendix

Table 5: Ablation study with flow-field diagnostics (generator components). Metrics are computed on 81-frame future prediction. Lower is better (

\downarrow

). All results use Euler integration with 10 steps. w/o fusion removes the token-aligned history fusion module; w/o anchoring disables boundary anchoring of the first future latent slice ; 20k steps is an early flow-matching checkpoint of the generator; w/o on-policy matches the full flow-matching training budget of our method but omits the on-policy

K

-step fine-tuning stage.

Metric	Method	Kinetics	RoboTAP	Kubric	MagicData (E)
FVMD $\downarrow$	WHN (L)	8999	7587	4872	10383
	WHN (L) ∗	8755	5855	6284	9691
	Ours	3626	2467	1338	2968
	Ours ∗	3182	2546	1724	2358
FlowTV ( $10^{2}$ ) $\downarrow$	WHN (L)	24.71	27.26	12.56	26.74
	WHN (L) ∗	22.06	14.46	18.41	14.46

Appendix

Table 6: Effect of sampler choice and step count. Metrics are computed on 81-frame future prediction; lower is better (

\downarrow

). Entries marked with ∗ use Dormand–Prince (DOPRI5) with 100 steps, while unmarked entries use Euler with 10 steps (default).

Explore similar work

Learning Long-term Motion Embeddings for Efficient Kinematics Generation
Apr 13, 2026Nick Stracke, Kolja Bauer, Stefan Andreas Baumann +3Motion Representation Learning Human Motion Generation
Munich Center for Machine Learning · Apple
Envisioning the Future, One Step at a Time
Apr 10, 2026Stefan Andreas Baumann, Jannik Wiese, Tommaso Martorella +2Trajectory Prediction Autoregressive Diffusion
Munich Center for Machine Learning (MCML) · Netflix
Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution
Oct 2, 2026Feiran Wang, Bin Duan, Junyi Wu +2Video Generation Controllable Video Generation
University of Illinois at Chicago · University of Michigan Ann Arbor · Cisco

Apr 13, 2026J/K move · Enter open · S save

Learning Long-term Motion Embeddings for Efficient Kinematics Generation

Nick Stracke, Kolja Bauer, Stefan Andreas Baumann, Miguel Angel Bautista+2

Munich Center for Machine Learning · Apple

Motion Representation Learning Human Motion Generation

Open paper AskPDF arXiv

Understanding and predicting motion is a fundamental component of visual intelligence. Although modern video models exhibit strong comprehension of scene dynamics, exploring multiple possible futures through full video synthesis remains prohibitively inefficient. We model scene dynamics orders of magnitude more efficiently by directly operating on a long-term motion embedding that is learned from large-scale trajectories obtained from tracker models. This enables efficient generation of long, realistic motions that fulfill goals specified via text prompts or spatial pokes. To achieve this, we first learn a highly compressed motion embedding with a temporal compression factor of 64x. In this space, we train a conditional flow-matching model to generate motion latents conditioned on task descriptions. The resulting motion distributions outperform those of both state-of-the-art video models and specialized task-specific approaches.

Abstract

Explore similar work

Learning Long-term Motion Embeddings for Efficient Kinematics Generation

Envisioning the Future, One Step at a Time

Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution