cs.LGNov 13, 2026

Joint World Action Models through Differentiable Trajectory Optimization

Authors: Mintae Kim, Koushil Sreenath

Organizations: Hybrid Robotics, BAIR, UC Berkeley

Abstract

Joint world action models (WAMs) generate future actions and states together, modeling world-action trajectories rather than actions alone. However, trajectory imitation does not ensure that these trajectories are task-effective or consistent with dynamics. We present world action trajectory optimization (WATO), which optimizes generated actions and future states jointly for control. WATO uses both as references in a dynamics-constrained trajectory optimization problem and differentiates the task loss through the optimized trajectory to train the generator. This makes generated trajectories task-aware and dynamics-consistent while using trajectory optimization only during training. At deployment, WATO directly generates candidate trajectories and ranks them with a world model, without iterative trajectory optimization. Across 77 visual control environments, WATO improves average success from 79.9%79.9\% for the imitation joint WAM to 91.7%91.7\%, outperforming strong model-based, goal-conditioned RL, and generative control baselines. Compared with cross-entropy method (CEM), WATO is on average 11.5×11.5\times faster and uses 600.5×600.5\times fewer world model evaluations at deployment.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. What Limits Cross-Task Generalization in World Action Models?

    Oct 12, 2026Chen Zhao, Zipeng Ye, Yihan Lin +2World Action ModelsWorld Models for Robotics

  2. Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation

    Sep 29, 2026Xiangcheng Zhan, Zirui Chen, Yicheng Zhao +2Test-Time AdaptationWorld Model Learning

  3. World-Action Models for Robot Learning and Control: A Survey

    Sep 13, 2026Zuxing Lu, Hongjia Zhai, Guanzhi Wang +13World Action ModelsAction-Conditioned World Models