Understanding and predicting motion is a fundamental component of visual intelligence. Although modern video models exhibit strong comprehension of scene dynamics, exploring multiple possible futures through full video synthesis remains prohibitively inefficient. We model scene dynamics orders of magnitude more efficiently by directly operating on a long-term motion embedding that is learned from large-scale trajectories obtained from tracker models. This enables efficient generation of long, realistic motions that fulfill goals specified via text prompts or spatial pokes. To achieve this, we first learn a highly compressed motion embedding with a temporal compression factor of 64x. In this space, we train a conditional flow-matching model to generate motion latents conditioned on task descriptions. The resulting motion distributions outperform those of both state-of-the-art video models and specialized task-specific approaches.
Figures & tables
Figure 2: Our approach to learn a dense motion space. Sparse tracker trajectories and the start frame are encoded into a latent motion grid, which enables dense reconstruction at arbitrary spatial query points. The model jointly attends over trajectory tokens and frame features, producing temporally consistent, spatially dense motion predictions.
Figure 3: Model architecture to generate in learned motion space. We train a conditional flow matching model that learns a vector field over latent motion grids. We condition on either pokes [ 7 ] or text prompts, enabling controllable and semantically coherent motion synthesis in the learned motion space. The frame f0 provides context over the scene.
Figure 4: Temporal compression enables our model to generate plausible motions more efficiently. Under a fixed compute budget, both motion generation quality and inference throughput improve substantially with stronger compression (left), with only a minor reduction in reconstruction fidelity (middle). We attribute these gains to the reduced token count, which improves training efficiency, and to an increasingly semantic latent structure, as evidenced by higher kNN retrieval accuracy (right).
Figure 5: Example of multiple plausible motion hypotheses for the flight of an eagle, generated by our model from a single start frame. Our model produces diverse, physically coherent motions even in complex natural scenes, illustrating the expressiveness of the learned motion space.
Figure 6: Qualitative examples demonstrating diverse motion reasoning capabilities. These results highlight that the model captures semantic motion structure and generalizes to varied motion regimes.
Figure 7: LIBERO rollout samples. Our track predictor forecasts tracks 16 steps ahead (visualized), enabling long-horizon planning. A policy head conditions on these predictions to select the next actions, with predictions updated after every new observation.
Model
LIBERO succ. rates ↑
10
90
Spatial
Goal
Object
Avg.
ATM [ 46 ]
39.3
48.4
68.5
77.8
68.0
60.4
Amplify [ 11 ]
62.0
66.0
69.0
75.0
85.0
71.4
ZipMo (Ours)
66.3
62.3
91.3
69.7
98.0
77.5
Tra-MoE [ 48 ]
28.5
–
62.5
81.0
73.5
61.4
ZipMo (Ours)
61.0
–
85.7
82.3
92.0
80.3
Table 2: Success rates on LIBERO tasks following the respective training and evaluation setup of ATM [ 46 ] (top) and of Tra-MoE [ 48 ] (bottom).
Model
Time ↓
Min MSE ↓
Mean MSE ↓
EPE ↓
Wan [ 44 ]
1h
28.67
57.02
4.68
Veo 3 [ 13 ]
?
36.18
94.00
6.21
ZipMo (Ours)
1s
27.08
39.53
1.17
Table 3: Samples Matched: We sample k=8 times from each model and track the generated videos to report our distributional metrics as well as conditioning adherence with EPE. This is an unfavorable setting for us, as our model is much smaller. Still, we outperform the video while being orders of magnitude faster.
Model
Samples ↑
Min MSE ↓
Mean MSE ↓
EPE ↓
Wan [ 44 ]
1
64.20
64.20
5.23
Veo 3 [ 13 ]
1
65.99
65.99
5.84
ZipMo (Ours)
>10k
21.29
40.33
1.17
Table 4: Time Matched: Setup similar to Tab. 3 but now matching wall clock time for sampling. Our performance lead increases drastically due to the efficiency of our approach.
Model
LIBERO-90
LIBERO-10
Side
Effector
Side
Effector
Discriminative
ATM [ 46 ]
47.82
123.01
59.10
131.58
Tra-MoE [ 48 ]
39.77
116.47
50.37
131.75
Generative
WHNSingle [ 9 ] *
17.89
57.64
26.18
63.47
Table E : Comparison on text-conditioned trajectory prediction on the LIBERO [ 27 ] dataset, measured by MSE and following the evaluation setup of [ 9 ] . Numbers marked with an asterisk * are taken directly from the original paper, as no official checkpoints or training code were released. All other results were reproduced using the official checkpoints. The upper part of the table reports regression-based methods, while the lower part compares generative methods. The Single metric is the MSE for the first sample. All distributional metrics are computed over k=8 samples per trajectory.
Model
LIBERO success rates ↑
10
90
Spatial
Goal
Object
Avg.
ATM [ 46 ]
39.3
48.4
68.5
77.8
68.0
60.4
Amplify [ 11 ]
62.0
66.0
69.0
75.0
85.0
71.4
ZipMo (Ours)
66.3 ± 4.0
62.3 ± 0.4
91.3 ± 0.6
69.7 ± 2.5
98.0 ± 2.6
77.5 ± 1.8
Tra-MoE [ 48 ]
28.5
–
62.5
81.0
73.5
61.4
ZipMo (Ours)
61.0 ± 1.7
–
85.7 ± 2.1
82.3 ± 3.5
92.0 ± 2.0
80.3 ± 1.4
Table F : Same setup as Table 2 , with seed-wise variability reported for our method. For each experiment, we evaluate three random seeds, with 10 rollouts per task per seed, and report mean success rate ± standard deviation.
Train \ Eval
TapNext
CoTracker3
TapNext
96.8
97.0
CoTracker3
96.3
97.3
Table H : Cross-tracker reconstruction accuracy δavg .
Ablation
δavg↑
Ours
96.8
+ track dropout
94.0
+ filter occlusions
93.2
Table I : Reconstruction accuracy degradation under systematic tracker degradation during training.
Model
Min MSE ↓
Mean MSE ↓
EPE ↓
DAVIS [ 32 ]
Motion I2V [ 34 ]
222.2
307.0
16.37
ZipMo (Ours)
155.1
233.0
0.83
PhysicsIQ [ 29 ]
Motion I2V [ 34 ]
177.8
225.1
12.4
ZipMo (Ours)
90.60
143.65
0.76
Table J : Dense Track prediction results on Davis (top) and PhysicsIQ (bottom) . We condition both models on the maximum number of pokes from the start to the end frames.
Figure H : Additional qualitative samples. The first image in every column is the prompt, with arrows indicating pokes, i.e., where a specific start position should end. The remaining four images are samples from our models, illustrating trajectories of randomly selected query points. The color gradient of the trajectories acts as an indicator of how fast the motion is happening. The gradient starts at blue, moves to green, yellow, and finally ends at red.
Figure I : Additional qualitative samples. The first image in every column is the prompt, with arrows indicating pokes, i.e., where a specific start position should end. The remaining four images are samples from our models, illustrating trajectories of randomly selected query points. The color gradient of the trajectories acts as an indicator of how fast the motion is happening. The gradient starts at blue, moves to green, yellow, and finally ends at red.
Figure J : We ablate the effect of arranging latent tokens in a grid using 2D RoPE. We measure PCK using a 1 pixel threshold and report reconstruction performance for encoded positions (AE setting), as well as unseen positions (MAE setting). We find that providing positional information slightly increases performance.
Figure K : Motion Generation performance (Min MSE) across different numbers of function evaluations (NFEs). Trajectories are densely conditioned on target positions, sampled k=128 times per start frame.
Figure L : Our model enables dense motion inference from sparse tracker inputs by converting observed tracks (pokes) into conditioning signals that specify desired endpoints in the final frame. The visualizations show dense flow toward the goal frame, illustrating how our approach infers globally coherent motion. Only the endpoint for each poke is provided; the resulting dense prediction demonstrates our model’s ability to interpolate and inpaint plausible trajectories across the entire scene.
Figure M : Motion generation quality over training for different temporal compression factors, measured against wall-clock time to account for varying computational cost. Matching step time (left) and batch size (right) both show that higher temporal compression yields more efficient learning and faster emergence of realistic motion.
Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at https://trajloom.github.io/.
Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu +4
McMaster University · University of British Columbia · Vector Institute +2
Video world models aim to preserve scene structure and predict how dynamic objects evolve beyond visual observations. We present Kepler4D, a framework for future video generation through explicit 4D scene state evolution. Given a monocular video, Kepler4D constructs a shared 3D representation of background geometry, object motion histories, coarse spatial supports, and semantic context. Chain-of-Motion summarizes observed motion and uses a vision-language model to select structured speed and heading decisions and decide whether to bound object-center height from below. A deterministic rollout converts these decisions into future object trajectories for inspection and editing before synthesis. We render the evolving proxies into geometric controls for a pretrained video generator, separating coarse object motion from the synthesis of appearance and articulation. Experiments on real-world videos demonstrate that Kepler4D enables controllable object motion and plausible future rollout while preserving scene consistency.
Feiran Wang, Bin Duan, Junyi Wu +2
University of Illinois at Chicago · University of Michigan Ann Arbor · Cisco
Text-guided long-term human motion generation is essential for applications such as continuously operating digital avatars. Despite recent progress on short clips, existing models struggle with multi-segment instructions that involve anaphora, where constraints established in earlier segments must be preserved over time. In this paper, we introduce a novel task of long-term motion generation from such anaphoric compositions. To mitigate this, we present ELMA, an episodic long-term motion dataset with dense segment-level annotations that preserve cross-segment dependencies. Built via a fully automated collection and filtering pipeline, ELMA extracts high-quality, continuous 30-second clips from in-the-wild videos, capturing the episodic narratives necessary for long-range state modeling. Leveraging this dataset, we fine-tune state-of-the-art motion generation models to incorporate long-range context. Extensive experiments reveal fundamental shortcomings in current models when handling context-dependent instructions. We demonstrate that ELMA enables significant improvements in both global physical realism and semantic alignment, setting a robust new baseline for long-term motion synthesis.
Shun Kato, Dixin Yang, Kotaro Amaya +5
Keio University · Keio AI Center · Sony Group Corporation