cs.CVApr 13, 2026
SaveLearning Long-term Motion Embeddings for Efficient Kinematics Generation
Organizations: Munich Center for Machine Learning · Apple
Abstract
Understanding and predicting motion is a fundamental component of visual intelligence. Although modern video models exhibit strong comprehension of scene dynamics, exploring multiple possible futures through full video synthesis remains prohibitively inefficient. We model scene dynamics orders of magnitude more efficiently by directly operating on a long-term motion embedding that is learned from large-scale trajectories obtained from tracker models. This enables efficient generation of long, realistic motions that fulfill goals specified via text prompts or spatial pokes. To achieve this, we first learn a highly compressed motion embedding with a temporal compression factor of 64x. In this space, we train a conditional flow-matching model to generate motion latents conditioned on task descriptions. The resulting motion distributions outperform those of both state-of-the-art video models and specialized task-specific approaches.
Figures & tables
Figure 2: Our approach to learn a dense motion space. Sparse tracker trajectories and the start frame are encoded into a latent motion grid, which enables dense reconstruction at arbitrary spatial query points. The model jointly attends over trajectory tokens and frame features, producing temporally consistent, spatially dense motion predictions.
Figure 3: Model architecture to generate in learned motion space. We train a conditional flow matching model that learns a vector field over latent motion grids. We condition on either pokes [ 7 ] or text prompts, enabling controllable and semantically coherent motion synthesis in the learned motion space. The frame provides context over the scene.
Figure 4: Temporal compression enables our model to generate plausible motions more efficiently. Under a fixed compute budget, both motion generation quality and inference throughput improve substantially with stronger compression (left), with only a minor reduction in reconstruction fidelity (middle). We attribute these gains to the reduced token count, which improves training efficiency, and to an increasingly semantic latent structure, as evidenced by higher kNN retrieval accuracy (right).
Figure 5: Example of multiple plausible motion hypotheses for the flight of an eagle, generated by our model from a single start frame. Our model produces diverse, physically coherent motions even in complex natural scenes, illustrating the expressiveness of the learned motion space.
Figure 6: Qualitative examples demonstrating diverse motion reasoning capabilities. These results highlight that the model captures semantic motion structure and generalizes to varied motion regimes.
Figure 7: LIBERO rollout samples. Our track predictor forecasts tracks 16 steps ahead (visualized), enabling long-horizon planning. A policy head conditions on these predictions to select the next actions, with predictions updated after every new observation.
| Model | LIBERO succ. rates | |||||
| 10 | 90 | Spatial | Goal | Object | Avg. | |
| ATM [ 46 ] | 39.3 | 48.4 | 68.5 | 77.8 | 68.0 | 60.4 |
| Amplify [ 11 ] | 62.0 | 66.0 | 69.0 | 75.0 | 85.0 | 71.4 |
| ZipMo (Ours) | 66.3 | 62.3 | 91.3 | 69.7 | 98.0 | 77.5 |
| Tra-MoE [ 48 ] | 28.5 | – | 62.5 | 81.0 | 73.5 | 61.4 |
| ZipMo (Ours) | 61.0 | – | 85.7 | 82.3 | 92.0 | 80.3 |
Table 2: Success rates on LIBERO tasks following the respective training and evaluation setup of ATM [ 46 ] (top) and of Tra-MoE [ 48 ] (bottom).
| Model | Time | Min MSE | Mean MSE | EPE |
| Wan [ 44 ] | 1h | 28.67 | 57.02 | 4.68 |
| Veo 3 [ 13 ] | ? | 36.18 | 94.00 | 6.21 |
| ZipMo (Ours) | 1s | 27.08 | 39.53 | 1.17 |
Table 3: Samples Matched: We sample times from each model and track the generated videos to report our distributional metrics as well as conditioning adherence with EPE. This is an unfavorable setting for us, as our model is much smaller. Still, we outperform the video while being orders of magnitude faster.
| Model | Samples | Min MSE | Mean MSE | EPE |
| Wan [ 44 ] | 1 | 64.20 | 64.20 | 5.23 |
| Veo 3 [ 13 ] | 1 | 65.99 | 65.99 | 5.84 |
| ZipMo (Ours) | 21.29 | 40.33 | 1.17 |
Table 4: Time Matched: Setup similar to Tab. 3 but now matching wall clock time for sampling. Our performance lead increases drastically due to the efficiency of our approach.
| Model | LIBERO-90 | LIBERO-10 | ||
| Side | Effector | Side | Effector | |
| Discriminative | ||||
| ATM [ 46 ] | 47.82 | 123.01 | 59.10 | 131.58 |
| Tra-MoE [ 48 ] | 39.77 | 116.47 | 50.37 | 131.75 |
| Generative | ||||
| [ 9 ] * | 17.89 | 57.64 | 26.18 | 63.47 |
Table E : Comparison on text-conditioned trajectory prediction on the LIBERO [ 27 ] dataset, measured by MSE and following the evaluation setup of [ 9 ] . Numbers marked with an asterisk * are taken directly from the original paper, as no official checkpoints or training code were released. All other results were reproduced using the official checkpoints. The upper part of the table reports regression-based methods, while the lower part compares generative methods. The metric is the MSE for the first sample. All distributional metrics are computed over samples per trajectory.
| Model | LIBERO success rates | |||||
| 10 | 90 | Spatial | Goal | Object | Avg. | |
| ATM [ 46 ] | 39.3 | 48.4 | 68.5 | 77.8 | 68.0 | 60.4 |
| Amplify [ 11 ] | 62.0 | 66.0 | 69.0 | 75.0 | 85.0 | 71.4 |
| ZipMo (Ours) | 66.3 4.0 | 62.3 0.4 | 91.3 0.6 | 69.7 2.5 | 98.0 2.6 | 77.5 1.8 |
| Tra-MoE [ 48 ] | 28.5 | – | 62.5 | 81.0 | 73.5 | 61.4 |
| ZipMo (Ours) | 61.0 1.7 | – | 85.7 2.1 | 82.3 3.5 | 92.0 2.0 | 80.3 1.4 |
Table F : Same setup as Table 2 , with seed-wise variability reported for our method. For each experiment, we evaluate three random seeds, with 10 rollouts per task per seed, and report mean success rate standard deviation.
| Train Eval | TapNext | CoTracker3 |
| TapNext | 96.8 | 97.0 |
| CoTracker3 | 96.3 | 97.3 |
Table H : Cross-tracker reconstruction accuracy .
| Ablation | |
| Ours | 96.8 |
| track dropout | 94.0 |
| filter occlusions | 93.2 |
Table I : Reconstruction accuracy degradation under systematic tracker degradation during training.
| Model | Min MSE | Mean MSE | EPE |
| DAVIS [ 32 ] | |||
| Motion I2V [ 34 ] | 222.2 | 307.0 | 16.37 |
| ZipMo (Ours) | 155.1 | 233.0 | 0.83 |
| PhysicsIQ [ 29 ] | |||
| Motion I2V [ 34 ] | 177.8 | 225.1 | 12.4 |
| ZipMo (Ours) | 90.60 | 143.65 | 0.76 |
Table J : Dense Track prediction results on Davis (top) and PhysicsIQ (bottom) . We condition both models on the maximum number of pokes from the start to the end frames.
Figure H : Additional qualitative samples. The first image in every column is the prompt, with arrows indicating pokes, i.e., where a specific start position should end. The remaining four images are samples from our models, illustrating trajectories of randomly selected query points. The color gradient of the trajectories acts as an indicator of how fast the motion is happening. The gradient starts at blue, moves to green, yellow, and finally ends at red.
Figure I : Additional qualitative samples. The first image in every column is the prompt, with arrows indicating pokes, i.e., where a specific start position should end. The remaining four images are samples from our models, illustrating trajectories of randomly selected query points. The color gradient of the trajectories acts as an indicator of how fast the motion is happening. The gradient starts at blue, moves to green, yellow, and finally ends at red.
Figure J : We ablate the effect of arranging latent tokens in a grid using 2D RoPE. We measure PCK using a 1 pixel threshold and report reconstruction performance for encoded positions (AE setting), as well as unseen positions (MAE setting). We find that providing positional information slightly increases performance.
Figure K : Motion Generation performance (Min MSE) across different numbers of function evaluations (NFEs). Trajectories are densely conditioned on target positions, sampled times per start frame.
Figure L : Our model enables dense motion inference from sparse tracker inputs by converting observed tracks (pokes) into conditioning signals that specify desired endpoints in the final frame. The visualizations show dense flow toward the goal frame, illustrating how our approach infers globally coherent motion. Only the endpoint for each poke is provided; the resulting dense prediction demonstrates our model’s ability to interpolate and inpaint plausible trajectories across the entire scene.
Figure M : Motion generation quality over training for different temporal compression factors, measured against wall-clock time to account for varying computational cost. Matching step time (left) and batch size (right) both show that higher temporal compression yields more efficient learning and faster emergence of realistic motion.