Joint World Action Models through Differentiable Trajectory Optimization
Organizations: Hybrid Robotics, BAIR, UC Berkeley
Abstract
Joint world action models (WAMs) generate future actions and states together, modeling world-action trajectories rather than actions alone. However, trajectory imitation does not ensure that these trajectories are task-effective or consistent with dynamics. We present world action trajectory optimization (WATO), which optimizes generated actions and future states jointly for control. WATO uses both as references in a dynamics-constrained trajectory optimization problem and differentiates the task loss through the optimized trajectory to train the generator. This makes generated trajectories task-aware and dynamics-consistent while using trajectory optimization only during training. At deployment, WATO directly generates candidate trajectories and ranks them with a world model, without iterative trajectory optimization. Across visual control environments, WATO improves average success from for the imitation joint WAM to , outperforming strong model-based, goal-conditioned RL, and generative control baselines. Compared with cross-entropy method (CEM), WATO is on average faster and uses fewer world model evaluations at deployment.
Figures & tables
| Goal-conditioned learning | Generative control | Model-based control | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Environment | GCBC | GCIVL | GCIQL | QRL | Flow Policy | Joint WAF | GPC | CEM | Grad. MPC | WATO |
| TwoRooms | ||||||||||
| Reacher | ||||||||||
| PushT | ||||||||||
| cube-single | ||||||||||
| cube-double | ||||||||||
| Environment | Before WATO | After WATO |
|---|---|---|
| TwoRooms | ||
| Reacher | ||
| PushT | ||
| cube-single | ||
| cube-double | ||
| scene-play |
| Environment | Latency | WM evals |
|---|---|---|
| TwoRooms | ||
| Reacher | ||
| PushT | ||
| cube-single | ||
| cube-double | ||
| scene-play |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| GCIQL | WATO: pretrained LeWM | |||
|---|---|---|---|---|
| Environment | Pretrained LeWM | End-to-end training | ||
| TwoRooms | ||||
| Reacher | ||||
| PushT | ||||
| cube-single | ||||
| cube-double | ||||
| Environment | Source | Selected checkpoint | ||
|---|---|---|---|---|
| TwoRooms | Released LeWM | Released | ||
| Reacher | Released LeWM | Released | ||
| PushT | Released LeWM | Released | ||
| cube-single | Released LeWM | Released | ||
| cube-double | Trained in this work | updates | ||
| scene-play | Trained in this work | updates |
| Setting | Joint WAF training | WATO training |
|---|---|---|
| Optimizer | AdamW | AdamW |
| Weight decay | ||
| Learning rate | ||
| Batch size | (FM), (task) | |
| Updates | ||
| Warmup | updates | updates |
| Metric | Value |
|---|---|
| Joint WAF pretraining | h |
| WATO training | h |
| Total pipeline | h |
| WATO / Joint WAF | |
| Pipeline / Joint WAF-only | |
| FM-only continuation | h |
| Environment | Success criterion |
|---|---|
| TwoRooms | The Euclidean distance between the agent position and target position is less than coordinate units. |
| Reacher | The maximum absolute joint-position error over the two target joints is less than radians. |
| PushT | The positional error over the first four task-state coordinates is less than , and the wrapped block-orientation error is less than radians. |
| cube-single | The Euclidean distance between the cube position and target position is at most meters. |
| cube-double | Both cubes are within meters of their respective targets, i.e., . |
| scene-play | The cube-position error is at most meters, the absolute drawer and window position errors are each at most , and every discrete button state matches its target state. All conditions must hold simultaneously. |
| Training objective | Reference design | |||||||
|---|---|---|---|---|---|---|---|---|
| Environment | Joint WAF | Direct | Direct + Consistency | Distillation | Action only | State only | Rollout-derived reference | WATO |
| TwoRooms | ||||||||
| Reacher | ||||||||
| PushT | ||||||||
| cube-single | ||||||||
| cube-double | ||||||||
| Environment | |||||
|---|---|---|---|---|---|
| PushT | |||||
| cube-single | |||||
| Before WATO | After WATO | |||
|---|---|---|---|---|
| Environment | Joint WAF | + Traj. Opt. | WATO | + Traj. Opt. |
| TwoRooms | ||||
| Reacher | ||||
| PushT | ||||
| cube-single | ||||
| cube-double | ||||
| Environment | GCBC | GCIVL | GCIQL | QRL | Flow Policy |
|---|---|---|---|---|---|
| TwoRooms | |||||
| Reacher | |||||
| PushT | |||||
| cube-single | |||||
| cube-double | |||||
| scene-play |
| Environment | CEM (s) | WATO (s) |
|---|---|---|
| TwoRooms | ||
| Reacher | ||
| PushT | ||
| cube-single | ||
| cube-double | ||
| scene-play |
| Method | Count | Relative to WATO |
|---|---|---|
| Joint WAF | ||
| CEM | ||
| Grad. MPC | ||
| WATO |
World Action Models
World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.
See all 240 papers