FRAPPE: Infusing World Modeling into Generalist Policies via Multiple Future Representation Alignment
Authors: Han Zhao, Jingbo Wang, Wenxuan Song, Shuai Chen, Yang Liu, Yan Wang, Haoang Li, Donglin Wang
Organizations: Zhejiang University · Westlake University · HKUST (GZ) · South China University of Technology · ShanghaiTech University · Tsinghua University
Enabling VLA models to predict environmental dynamics, known as world modeling, has been recognized as essential for improving robotic reasoning and generalization. However, current approaches face two main issues: 1. The training objective forces models to over-emphasize pixel-level reconstruction, which constrains semantic learning and generalization 2. Reliance on predicted future observations during inference often leads to error accumulation. To address these challenges, we introduce Future Representation Alignment via Parallel Progressive Expansion (FRAPPE). Our method adopts a two-stage fine-tuning strategy: In the mid-training phase, the model learns to predict the latent representations of future observations; In the post-training phase, we expand the computational workload in parallel and align the representation simultaneously with multiple different visual foundation models. By significantly improving fine-tuning efficiency and reducing dependence on action-annotated data, FRAPPE provides a scalable and data-efficient pathway to enhance world-awareness in generalist robotic policies. Experiments on the RoboTwin benchmark and real-world tasks demonstrate that FRAPPE outperforms state-of-the-art approaches and shows strong generalization in long-horizon and unseen scenarios.
Figures & tables
Figure 1: We demonstrate that FRAPPE significantly outperforms the state-of-the-art models in both simulated and real-world complex scenarios, and it can effectively leverage data from different levels of the training data pyramid.
Figure 2: Overview of training and inference. During the training phase, the model progressively learns to align with the representation spaces of multiple visual foundation models simultaneously. The model is trained through a two-stage training process to extends to parallel processing of multiple input streams while aligning diverse visual representations. Similarly, parallel inference is implemented during the inference stage.
Method
Handover Mic
Pick Dual Bottles
Handover Block
Place Object Basket
Easy
Hard
Easy
Hard
Easy
Hard
Easy
Hard
DP
53.0%
0.0%
24.0%
0.0%
10.0%
0.0%
15.0%
0.0%
VPP
91.0%
7.0%
28.0%
6.0%
8.0%
0.0%
30.0%
0.0%
RDT
90.0%
31.0%
45.0%
14.0%
45.0%
14.0%
30.0%
6.0%
π0
98.0%
13.0%
57.0%
12.0%
45.0%
8.0%
16.0%
2.0%
π0.5
98.0%
13.0%
37.0%
13.0%
2.0%
0.0%
42.0%
4.0%
Table 1: Comparisons with state-of-the-art methods on RoboTwin 2.0 benchmark. Please note that Bold and Underline denote the best and the second-best performances among the methods.
Paradigm
Method
Training Steps
Success Rates (%)
No.
Easy
Hard
Average
0
RDT
20k
59.0
20.5
39.8
1
mid-train (full ft)
20k
63.0
27.5
45.3
2
mid-train (prefix & lora ft)
20k
48.0
8.5
28.3
3
post-train (prefix ft)
20k
25.0
4.0
14.5
4
post-train (prefix & lora ft)
20k
46.0
9.0
27.5
Table 2: Comparison of training paradigms. Please note that Bold and Underline denote the best and the second-best performances, respectively, across the paradigms. The results indicate that both mid-training and post-training phases are significant for the method’s improvement.
RDT (5 steps)
mid-train (5 steps)
post-train (5 steps)
post-train (3 steps)
Inference Memory (GB)
3.7
3.7
8.0
8.0
Latency (Sec)
0.214
0.228
0.235
0.173
Success Rates (%)
39.8
45.3
52.3
48.5
Table 3: Comparison of Inference Efficiency
Figure 3: Experiments on different parameter scales on 4 tasks on the RoboTwin 2.0. The 130M backbone model fine-tuned using either LoRA or full-parameter fine-tuning under the FRAPPE consistently outperforms the naively fine-tuned RDT-130M across all tasks and remains competitive when compared with the naively fine-tuned RDT-1B. Especially in the Stack Bowls Two-Hard task, the improvement is significant.
Figure 4: Real-world experiment results in seen and unseen scenarios. We evaluate our FRAPPE and prior SOTA VLAs on 4 representative tasks, each with different axes of generalization. Seen means the settings were included in the training data, while Unseen refers to new task settings that the model did not encounter during training.
Figure 5: Long-horizon Performance. Each data point represents the success rate of completing up to and including that corresponding subtask.
Figure 6: Leveraging human egocentric data without action labels. (a). Experimental targets, including 5 objects that are easy to pick up, and 5 objects that are difficult to pick up. (b). Multi-task results of FRAPPE trained on Robot (task) and + Ego (web) across 8 target object categories. Here, “multi-task” denotes that one model is used to be trained and tested on multiple target objects. (c). The results of FRAPPE trained on Robot (task) , + Ego (web) and + Ego (task) + Ego (web) data mixture, respecitvely.
Robotic world models predict action-conditioned future scenes, providing a foundation for understanding action outcomes. However, turning alternative predictions into useful learning signals remains challenging: similar candidates limit informative quality comparisons, while diverging trajectories require persistent maintenance of their individual histories. We introduce FutureWorlds, a framework that unifies candidate construction, history maintenance, and learning from relative quality. Built on a multimodal discrete autoregressive model, FutureWorlds uses diverse beam search during reinforcement learning to construct candidate futures that balance confidence and diversity. Candidate-specific bounded memory preserves scene states and ensures that generation and policy scoring use matching histories. We further propose MemSPO (Memory-Conditioned Search-Guided Policy Optimization), which converts video trajectory rewards into group-relative advantages to optimize the world model. On RT-1, BridgeV2, and RoboCasa, FutureWorlds reduces LPIPS for 32-frame predictions by 14.78%, 20.84%, and 9.12%, respectively, relative to the strongest baseline on each dataset. Under fixed evaluation configurations, only 200 MemSPO updates further improve generation quality and support continued prediction beyond the training horizon. Memory ablations, decoding sensitivity analysis, and optical-flow evaluation show that these gains extend beyond visual quality to more accurate motion prediction and more consistent object states. Project page and code: https://github.com/Alexander-wu/FutureWorlds.
Vision-Language-Action (VLA) models map observations to actions with no objective that accounts for how the world responds, so their robustness is bounded primarily by data coverage. World models carry precisely that missing objective and are better grounded for it, yet rolling the future forward costs seconds per decision and rules them out of the control loop. We show the two can be separated. What a world model knows about physical scenes lives in its internal features; generating the future is merely the objective that produced them, so the grounding can be inherited while the generative machinery is left behind. We add one feature-alignment term to ordinary VLA training: a frozen world model is run over the training frames once and cached, and the student learns to agree with that cache. No teacher is loaded during training, the projector is discarded after it, and the deployed policy is identical to the undistilled baseline, running in 32ms and 1.86GB on a consumer RTX5090, so every gain is attributable to the representation rather than to added capacity or test-time compute. A 0.8B student reaches 97.9% on LIBERO, improves from 48.2% to 50.5% on RoboCasa-GR1 humanoid manipulation, and the same objective carries over to real hardware, on both a single-arm and a bimanual platform. The gain survives changes of student scale, backbone, alignment layer, and teacher, indicating a broad representational prior rather than a fragile alignment between two particular networks. Project page: https://thaw-vla.trung-dt.com/.
Trung Dao, Sankalp Yamsani, Jaden Park +2
University of Wisconsin–Madison, USA. · University of Illinois Urbana-Champaign, USA.
Vision-language-action (VLA) policies excel at generalist robotic manipulation but generate actions without explicitly modeling their environmental consequences. World action models (WAMs) incorporate future prediction, yet most rely on costly video generation, while latent alternatives either use predicted futures only as auxiliary targets or infer latent actions from suboptimal visual transitions through costly large-scale video pretraining. We introduce WAVE-0 (World-model-Augmented VLA Execution), a framework that bridges VLAs and WAMs through a lightweight latent world model. By connecting the semantic understanding of pretrained VLAs with explicit future prediction, WAVE-0 allows anticipated action consequences to directly guide action generation. Built on a pretrained VLA, its 257M world model translates VLM-generated latent action plans into predicted future representations that condition the action expert, unifying latent planning, future prediction, and action generation. The world model is trained directly on ground-truth robot actions with a counterfactual ranking objective, requiring neither pixel reconstruction nor large-scale video pretraining. WAVE-0 achieves success rates of 91.47% on RoboTwin 2.0 and 98.7% on LIBERO, along with 88.0%, 61.4%, and 75.0% on LIBERO-Plus, LIBERO-Pro, and LIBERO-Para, respectively. On real-world tasks, it improves the success rate of π0.5 from 38.3% to 73.3%. These results demonstrate strong manipulation performance and generalization across diverse distribution shifts.
Haomin Wang, Anqiong Ji, Ke Wang +4
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Baidu AI Cloud · Awaken Robotics +1