Organizations: Tsinghua University, Beijing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Harbin Institute of Technology, Harbin, China · Hangzhou Yunshenchu Technology Co., Ltd. (DEEP Robotics), Hangzhou, China
Manipulating moving objects requires a policy to anticipate contact events, yet vision-language-action (VLA) policies are commonly fine-tuned from the current observation alone. World action models (WAMs) learn predictive dynamics, but running a video-scale teacher or explicitly imagining future frames at deployment is costly. We introduce ForeTime-VLA, a dense pi0.5 policy that distills a future-aware, action-equivalent representation from a frozen Fast-WAM-derived teacher while remaining causal at inference. Offline, current and future video latents are compressed into a whitened 64-D target. Online, an eight-frame history encoder predicts this target together with manipulation phase and normalized time-to-transition. Four future tokens and one phase token condition the VLM prefix, while the predicted future and transition horizon condition the action expert. Training retains the original flow-matching action target and adds cosine, relational geometry, phase, time-to-transition, and action-equivalence objectives. On a deduplicated conveyor-belt dataset, we compare 40k-step checkpoints on 768 matched windows per split. Test MAE decreases from 0.134119 to 0.130593 (2.63%; paired-bootstrap 95% CI: 0.82-4.48% improvement), and test L2 decreases by 3.02%, at a 2.46-2.93% latency cost. In quantitative real-robot evaluation, ForeTime-VLA achieves 81.1% stationary and 58.9% slow-moving grasp success, exceeding the next-best reference by 12.2 and 22.2 percentage points, respectively. Across three belt speeds, it completes 44/90 grasps versus 23/90 for pi0.5, including 11/30 versus 2/30 at fast speed. The agreement between offline orientation gains and reduced real-robot contact-pose failures supports causal future-token distillation as an effective way to improve dynamic manipulation without deploying the world-model teacher.
Figures & tables
Fig. 1: ForeTime-VLA overview. Left: the offline teacher encodes the current observation and sampled future frames with a frozen video VAE, then maps them to a compact action-equivalent target. Center: the deployable student uses eight causal observations to predict future, phase, and transition horizon variables. These predictions condition both the slow π0.5 VLM prefix and the fast action-expert suffix, while the original noisy-action and target-flow construction is retained. Right: training combines action flow, future alignment, relational geometry, transition, phase, and auxiliary action-reconstruction losses. Future frames and the WAM are absent at test time.
Fig. 2: Offline action-equivalent future teacher. The frozen latent pipeline extracts current and future features (ct,ft) , and the adapter also receives robot state st . The privileged encoder uses (ct,ft,st) , whereas the causal branch uses (ct,st) . Both codes share an action decoder; the teacher additionally reconstructs the future feature difference. Teacher–causal alignment, variance/covariance regularization, and pairwise action geometry prevent collapse and retain action relevance. The selected 64-D teacher code is whitened and cached as ztT .
Fig. 3: Dataset structure and realized coverage. Left: the collection taxonomy spans stationary grasp, mobile tracking, navigate-then-grasp, and concurrent-grasp settings; outer labels give canonical episode counts. Upper right: episode and aligned-frame coverage by scene. Lower right: object composition, the duplicate-free split, and the recorded multimodal contract. Aggregate statistics exclude 28 exact duplicate files; the remaining 458 episodes contain 96,512 aligned frames and are split 402/31/25 for train/validation/test.
Split
Model
MSE
MAE
L2
Val.
π0.5 baseline
0.256338
0.138628
1.153725
ForeTime-VLA
0.249276
0.134522
1.128938
GR00T N1.6
0.278784
0.162035
1.285486
StarVLA
0.258356
0.163372
1.262874
SmolVLA
0.257650
0.147754
1.182720
Test
π0.5 baseline
0.248680
0.134119
1.104541
TABLE I: Offline MSE, MAE, and L2 over 768 matched windows per split. Lower is better. The lower panel reports paired relative improvement and 95% CIs for ForeTime-VLA versus the π0.5 baseline.
Split
Group
Baseline
ForeTime-VLA
Val
base XY
0.016555
0.016176 (+2.30%)
base yaw
0.008367
0.008367 (+0.00%)
EE position
0.023643
0.023479 (+0.69%)
EE RPY
0.478830
0.459112 (+4.12%)
arm
0.095593
0.094838 (+0.79%)
Test
base XY
0.015229
0.015642 ( −2.71 %)
TABLE II: Pooled action-group MAE. Parentheses show relative improvement.
Method
Overall MAE ↓
Transition MAE ↓
Transition EE-RPY MAE ↓
π0.5 baseline
0.122124
0.150140
0.501144
ForeTime-VLA w/o future condition
0.128787
0.159982
0.498349
ForeTime-VLA w/o phase/time condition
0.134846
0.160672
0.497727
ForeTime-VLA slow path only
0.136006
0.156587
0.494265
ForeTime-VLA fast path only
0.127140
0.161749
0.479751
ForeTime-VLA
0.118354
0.145320
0.482656
TABLE III: Offline inference-time component ablation. Overall metrics use 96 matched test windows; transition metrics use 48 matched phase-1 windows. Lower is better. All ForeTime-VLA variants share weights and differ only by component masking at inference.
Stationary grasp task
Method
Grasp success
π0.5
65.56%
GR00T N1.6
46.67%
StarVLA
68.89%
SmolVLA
63.33%
ForeTime-VLA
81.11%
TABLE IV: Real-robot outcomes for stationary and slow-speed moving grasp tasks. All entries are percentages; higher success and lower failure are better.
Fig. 4: Real-robot grasp completions across belt speeds. Successful grasps out of 30 trials; labels give completed/total trials.
Fig. 5: Qualitative real-robot grasp and phase prediction. Top: grasp of the moving target. Bottom: causal phase probabilities; at 9.8 s, grasp has 87.3% confidence before transport rises after acquisition.
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
Vision-Language-Action (VLA) models generalize across static manipulation but fail when objects move during task execution. They map the current observation to an action and assume the scene is stationary between observation and execution, so at any non-trivial object speed the resulting latency exceeds the time available to grasp. We close this gap with AHEAD (Anticipatory Horizon Extrapolation with Adaptive Dynamics), a predict-then-act wrapper that augments a frozen VLA with a motion-aware latent world model. A small world model trained on manipulation video forecasts future patch tokens in the VLA's feature space, conditioned on per-token velocity and acceleration from optical flow. A language-and-motion saliency mask concentrates prediction on task-relevant patches, and the model rolls forward for an adaptive horizon, halting when prediction uncertainty crosses a threshold. The frozen action decoder then receives the predicted future tokens in place of the current ones. AHEAD adds 4.9M parameters to a frozen 7B OpenVLA and reaches 79 to 97% success across 20 dynamic simulation scenarios where the strongest baseline reaches 31 to 58%. On a physical UFactory xArm 7, AHEAD succeeds on 29/30 to 30/30 on three conveyor and rolling-ball tasks, 23/30 on paddle interception, and 19/30 on projectile catching where every baseline scores 0/30.
Shahram Najam Syed, Arthur Jakobsson, Haoran Hao +1
Robotics Institute, Carnegie Mellon University, Pittsburgh, USA
Vision-Language-Action (VLA) models have recently incorporated world models to provide richer dynamic supervision beyond sparse action labels. However, explicitly predicting future images or videos may include control-irrelevant appearance, while guidance derived from holistic future visual representations and shared global action features may fail to establish timestep-specific correspondence between actions and local visual changes. To address this issue, we propose MotionWeave, a motion-centric future-dynamics framework for action-chunk prediction with two modules: the Action-Induced Motion Grounder (AIMG) and the Horizon Residual Composer (HRC). Specifically, AIMG conditions on action and proprioceptive representations to construct horizon-specific queries that localize interaction regions associated with each future action timestep from current visual tokens. HRC extracts differences between interaction representations at adjacent horizons, encodes them as temporal motion cues, and injects them into action tokens through a gated residual. During training, robot-arm masks rendered from future frames are used to construct KL-based motion-grounding supervision, while inference uses only the current observation. On six MetaWorld tasks, MotionWeave achieves a 75.3% average success rate, an absolute gain of 8.6% over π0 (66.7%), especially on sustained-interaction tasks. Our code is available at https://github.com/autu-mn/MotionWeave.
Jingqiu Wang, Yan Wang
School of Data Science and Engineering East China Normal University, Shanghai, China
A common approach to world-model simulation for vision-language-action (VLA) systems is to predict future RGB observations and then re-encode them into policy inputs, introducing an indirect interface between simulation and downstream policy execution. We instead investigate whether world dynamics can be modeled in a compact, policy-oriented state derived from VLM visual tokens. A key challenge is that raw VLM visual tokens are high-dimensional, making efficient and accurate autoregressive dynamics modeling challenging. To address this, we introduce Token-World, an action-conditioned world model that compresses VLM features into a compact token state, learns future dynamics in this reduced space, and maps predicted states back to the original policy-facing representation for downstream use. Across manipulation benchmarks, Token-World improves open-loop feature fidelity and policy-action consistency over recent world-model simulators, with slower degradation over long rollout horizons. In closed-loop evaluation, its simulated policy performance correlates more strongly with reference policy performance than Ctrl-World (r=0.794 vs.\ 0.583), while requiring lower simulation latency. Ablations further show that compact-representation design and dimensionality substantially affect future-state prediction. Code will be available at https://chuyaofu.github.io/Token-World/.
Chuyao Fu, Xiaowei Chi, Yuhan Rui +14
Southern University of Science and Technology · MUKA Robotics · Hong Kong University of Science and Technology +3