cs.ROFeb 25, 2026

World Guidance: World Modeling in Condition Space for Action Generation

Authors: Yue Su, Sijin Chen, Haixin Shi, Mingyu Liu, Zhengshen Zhang, Ningyuan Huang, Weiheng Zhong, Zhengbang Zhu, +2 more

Organizations: ByteDance Seed · The University of Hong Kong

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Leveraging future observation modeling to facilitate action generation presents a promising avenue for enhancing the capabilities of Vision-Language-Action (VLA) models. However, existing approaches struggle to strike a balance between maintaining efficient, predictable future representations and preserving sufficient fine-grained information to guide precise action generation. To address this limitation, we propose WoG (World Guidance), a framework that maps future observations into compact conditions by injecting them into the action inference pipeline. The VLA is then trained to simultaneously predict these compressed conditions alongside future actions, thereby achieving effective world modeling within the condition space for action inference. We demonstrate that modeling and predicting this condition space not only facilitates fine-grained action generation but also exhibits superior generalization capabilities. Moreover, it learns effectively from substantial human manipulation videos. Extensive experiments across both simulation and real-world environments validate that our method significantly outperforms existing methods based on future prediction. Project page is available at: https://selen-suyue.github.io/WoGNet/

Figures & tables

Figure 1: WoG first incorporates future observations into the action inference pipeline, projecting them into the condition space for action generation. Subsequently, it decouples future observations from the pipeline and simultaneously predicts these future conditions alongside actions, thereby transferring the knowledge of future conditions into the VLA model.
Figure 2: Overview of WoG . WoG is trained in two stages. In the first stage, future observations encoded by frozen vision foundation models are queried and compressed by a trainable Q-former-based Future Encoder to form condition representations, which, together with VLM-encoded current observations and instructions, are used for action prediction. In the second stage, the encoder and vision models are frozen, and the VLM backbone is trained to align with the conditions while predicting actions.
ModelVisual MatchingVariant AggregationOverall
Pick CokeMv NearDrawerAvg.Pick CokeMv NearDrawerAvg.Avg.
π0\pi_{0} [ pi0 ]72.7%65.3%38.3%58.8%75.2%63.7%25.6%54.8%56.8%
π0\pi_{0} -FAST [ fast ]75.3%67.5%42.9%61.9%77.6%68.2%31.3%59.0%60.5%
OpenVLA [ openvla ]16.3%46.2%35.6%32.7%54.5%47.7%17.7%39.8%33.8%
GR00T-N1 [ gr00t ]47.0%70.0%18.1%45.0%78.8%62.5%13.2%51.5%48.4%
Moto [ moto ]74.0%60.4%43.1%59.2%–––––
Table 1: SimplerEnv evaluation across different models on Google Robot tasks. Mv Near: Move Near, Drawer: Open/Close Drawer. WoG outperforms existing methods across the majority of tasks, with particularly notable gains in scenarios that necessitate efficient trajectory planning and collision avoidance.
ModelPut Spoon on TowelStack Green on YellowPut Carrot on PlatePut Eggplant in BasketOverall Average
Grasp SpoonSuccessGrasp G BlockSuccessGrasp CarrotSuccessGrasp EggplantSuccessGrasp Avg.Success Avg.
π0\pi_{0} [ pi0 ]45.8%29.1%25.0%0.0%50.0%16.7%91.6%62.5%40.1%27.1%
π0\pi_{0} -FAST [ fast ]62.5%29.1%58.5%21.9%54.0%10.8%83.3%66.6%48.3%32.1%
OpenVLA [ openvla ]4.1%0.0%33.0%0.0%12.5%0.0%8.3%4.1%7.8%1.1%
GR00T-N1 [ gr00t ]83.3%62.5%54.2%45.8%70.8%16.7%41.7%20.8%49.5%36.5%
UniVLA [ univla ]76.4%52.8%66.7%2.8%79.2%55.6%87.5%66.7%77.5%45.6%
Table 2: SimplerEnv evaluation across different models on WidowX Robot tasks. WoG outperforms existing methods across a broad spectrum of Pick-and-Place tasks, notably surpassing approaches that jointly model video prediction and latent actions.
ModelVisual MatchingVariant AggregationOverall
Pick CokeMv NearDrawerAvg.Pick CokeMv NearDrawerAvg.Avg.
WoG (dino)96.0%85.8%56.0%79.3%88.1%81.0%9.8%59.6%69.5%
WoG (dino-siglip)89.0%82.5%62.5%78.0%87.9%75.0%19.3%60.7%69.4%
WoG (dino-vae)97.7%87.1%61.1%82.0%89.7%77.5%12.4%59.9%70.9%
Table 3: SimplerEnv evaluation across different pretrained encoder configurations on Google Robot tasks. Mv Near: Move Near, Drawer: Open/Close Drawer.
ModelPut Spoon on TowelStack Green on YellowPut Carrot on PlatePut Eggplant in BasketOverall Average
Grasp SpoonSuccessGrasp G BlockSuccessGrasp CarrotSuccessGrasp EggplantSuccessGrasp Avg.Success Avg.
WoG (dino)83.3%66.7%41.7%8.3%50.0%29.2%100.0%91.7%68.8%49.0%
WoG (dino-siglip)95.8%79.2%75.0%33.0%70.8%50.0%100.0%91.7%85.4%63.5%
WoG (dino-vae)95.8%54.2%66.7%29.2%83.3%50.0%100.0%100.0%86.4%58.4%
Table 4: SimplerEnv evaluation across different different pretrained encoder configurations on WidowX Robot tasks.
ModelVisual MatchingVariant AggregationOverall
Pick CokeMv NearDrawerAvg.Pick CokeMv NearDrawerAvg.Avg.
WoG w/o Future Enc.93.3%79.2%52.8%75.1%87.3%72.9%14.8%58.3%66.7%
WoG w/o Future Enc. in Stage-II91.3%87.1%46.3%74.9%88.0%77.5%9.8%58.4%66.7%
WoG w. Future Enc.97.7%87.1%61.1%82.0%89.7%77.5%12.4%59.9%70.9%
Table 5: SimplerEnv evaluation across different Future Encoder ablation variants on Google Robot tasks. Mv Near: Move Near, Drawer: Open/Close Drawer.
ModelPut Spoon on TowelStack Green on YellowPut Carrot on PlatePut Eggplant in BasketOverall Average
Grasp SpoonSuccessGrasp G BlockSuccessGrasp CarrotSuccessGrasp EggplantSuccessGrasp Avg.Success Avg.
WoG w/o Future Enc.75.0%54.2%58.3%29.2%75.0%54.2%91.7%91.7%75.0%57.3%
WoG w/o Future Enc. in Stage-II83.3%70.8%58.3%29.2%50.0%37.5%95.8%91.7%71.8%57.3%
WoG w. Future Enc.95.8%54.2%66.7%29.2%83.3%50.0%100.0%100.0%86.4%58.4%
Table 6: SimplerEnv evaluation across different Future Encoder ablation variants on WidowX Robot tasks.
Figure 3: Overview of our real-world experiment setup . The figure shows our robotic platform and sensors (left), the execution of the three tasks under the in-distribution setup (middle), and the modifications applied for the out-of-distribution setup (right).
ModelMicrowavePick and PlaceFold the Towel
IDIDBackgroundNovel ObjectIDBackgroundLight ChangeNovel Object
UniVLA [ univla ]80%25%25% →\to 20%25% →\to 10%20%20% →\to 20%20% →\to 10%20% →\to 10%
VPP [ vpp ]90%55%55% →\to 30%55% →\to 15%45%45% →\to 30%45% →\to 20%45% →\to 30%
WoG100%60%60% →\to 55%60% →\to 40%60%60% →\to 50%60% →\to 35%60% →\to 50%
Table 7: Detailed performance comparison across all tasks. We report the success rate (%) for In-Distribution (ID) and various Out-of-Distribution scenarios. For OOD settings, scores are formatted as ID →\to OOD . Best results are highlighted in bold .
ModelMicrowavePick and PlaceFold the Towel
IDIDBackgroundNovel ObjectIDBackgroundLight ChangeNovel Object
Vanilla VLA90%45%45% →\to 45%45% →\to 40%40%40% →\to 25%40% →\to 10%40% →\to 30%
WoG w/o cotrain95%45%45% →\to 45%45% →\to 35%30%30% →\to 30%30% →\to 10%30% →\to 30%
WoG (Ours)100%60%60% →\to 55%60% →\to 40%60%60% →\to 50%60% →\to 35%60% →\to 50%
Table 8: Ablation of training stages in WoG. Vanilla VLA predicts actions solely from current observations. WoG w/o cotrain removes the condition supervision in the second stage. We report success rates (%); for OOD settings, scores are formatted as ID →\to OOD . Best results are highlighted in bold .
StrategyPick and PlaceFold the Towel
IDBackgroundNovel ObjectIDBackgroundLight ChangeNovel Object
w/o human data (Base)60%60% →\to 55%60% →\to 40%60%60% →\to 50%60% →\to 35%60% →\to 50%
w. human v.70%70% →\to 70%70% →\to 35%50%50% →\to 45%50% →\to 30%50% →\to 45%
w. human v./a.70%70% →\to 70%70% →\to 45%65%65% →\to 60%65% →\to 45%65% →\to 50%
Table 9: Performance under different human data integration strategies. w/o human data : trained solely on robot data. w. human v. : trained with unannotated human videos. w. human v./a. : trained with a mix of annotated and unannotated human videos. Best results are bold .
Figure 4: Performance after training with UMI data. Compared to training solely with robot data, the WoG showed a 42% improvement in performance on the P&P task and a 33% on the Fold task.
Figure 5: Detailed illustration of the query mechanisms within WoG. The left panel depicts the future encoder, which maintains a set of learnable queries to extract low-dimensional, action-relevant conditions from the features of pretrained vision models. The right panel illustrates the query mechanism for condition prediction during the second training stage. Here, a set of learnable query embeddings performs cross-attention with the last hidden states of the VLM, producing predictive representations that are supervised to align with the target conditions generated by the now-frozen future encoder.
DatasetRatioDatasetRatioDatasetRatio
Fractal0.140639Kuka0.140674Bridge0.146649
Taco Play0.032661Jaco Play0.005354Berkeley Cable Routing0.002907
Roboturk0.025753Viola0.010483Berkeley Autolab UR50.013452
Toto0.022367Stanford Hydra Dataset0.049202Austin Buds Dataset0.002343
NYU Franka Play Dataset0.009245Furniture Bench Dataset0.027112UCSD Kitchen Dataset0.000545
Austin Sailor Dataset0.024248Austin Sirius Dataset0.019224DLR EDAN Shared Control0.000613
Table 10: Dataset Sampling Ratios during the OXE pretraining.
Figure 6: Process of human manipulation process. We keep the same human manipulation data collection strategy as in [ gr3 , gr-dexter ] . Human trajectories can be efficiently collected with VR devices at a rate of approximately 450 trajectories per hour, substantially outpacing the teleoperated robot trajectory collection. Nevertheless, as discussed earlier, we use action annotations for only 11% of the data, while the remaining 89% are leveraged solely as unlabeled videos. We argue that this setting more faithfully reflects real-world conditions, where action-labeled human manipulation data are scarce but large-scale unlabeled videos are abundant, and is therefore essential for evaluating the scalability of WoG.
DatasetRatioDatasetRatioDatasetRatio
Fractal0.117016Kuka0.117045Bridge0.122016
Taco Play0.027175Jaco Play0.004455Berkeley Cable Routing0.002419
Roboturk0.021427Viola0.008722Berkeley Autolab UR50.011192
Toto0.018610Stanford Hydra Dataset0.040937Austin Buds Dataset0.001949
NYU Franka Play Dataset0.007692Furniture Bench Dataset0.022558UCSD Kitchen Dataset0.000454
Austin Sailor Dataset0.020175Austin Sirius Dataset0.015995DLR EDAN Shared Control0.000510
Table 11: Dataset Sampling Ratios for the Human Data Learning.

Explore similar work

CardsList

  1. WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction

    Oct 13, 2026Haomin Wang, Anqiong Ji, Ke Wang +4World Models for RoboticsWorld Action Models

    AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Baidu AI Cloud · Awaken Robotics +1

  2. World-Value-Action Model: Implicit Planning for Vision-Language-Action Systems

    Apr 16, 2026Runze Li, Hongyin Zhang, Junxi Jin +5Vision-Language-Action ModelsModel-Based Planning

    Westlake University Hangzhou, China · Nanjing University Suzhou Campus, 1520 Taihu Avenue, Suzhou, China

  3. PLaW-VLA: Predictive Latent World Modeling for Vision-Language-Action Policies

    Oct 8, 2026Yu Liu, Hetian Guo, Tianlv Huang +11Vision-Language-Action ModelsLatent World Models

    Jilin University · Astribot · Harbin Institute of Technology, Shenzhen +2