CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight

Chensheng Peng; Wenhao Ding; Ran Tian; Zewei Zhou; Jef Packer; Maximilian Igl; Peter Karkus; Yan Wang

cs.CVOct 1, 2026

CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight

Authors: Chensheng Peng, Wenhao Ding, Ran Tian, Zewei Zhou, Jef Packer, Maximilian Igl, Peter Karkus, Yan Wang, +4 more

Organizations: NVIDIA · UC Berkeley · UCLA

Abstract

World action models (WAMs) jointly predict actions (intent) and visual future (foresight). Standard training adds noise to recorded actions and video simultaneously, but such training paradigms introduce a mismatch: perturbed actions imply counterfactual future visual, while the noised video remains tied to the GT recording. In low-noise regime, the scene geometry and even the dynamic behavior remain clearly visible from the noisy future frames despite the added noise. We present CtrlWAM, which executes perturbed actions in a simulator and pairs them with their noised visual consequences for joint WAM learning. To accommodate the different denoising requirements of video and actions, we introduce warped video--action noise schedules that aim to keep visual layout responsive as action predictions evolve. We further extend the action interface from ego-only control to a variable number of agent streams, allowing a unified model to represent predicted or commanded futures for multiple agents. Driving experiments show more accurate action forecasts, closer agreement between generated video and actions, and better following of supplied commands; robotics experiments show stronger motion fidelity and controllability. Matched controls support the benefit of off-path renders for command following and manipulation fidelity. Together, these findings contribute to a more controllable world action model. Project page: https://ctrl-wam.github.io/

Figures & tables

Figure 1: Overview of CtrlWAM . Standard noising pairs perturbed actions with video tied to the recorded future. Aligned noising instead renders the consequences of the perturbed trajectories before adding visual noise, while retaining the recorded action–video pair as the training target. Combined with warped video–action noise schedules, the model supports joint prediction, action-conditioned video generation, video-conditioned action prediction, and control of surrounding agents, with applications to driving and robotic manipulation.

Figure 2: Safe video, unsafe action. This schematic illustrates intent–foresight misalignment: after noising and denoising, the video depicts a safe pass, while a small deviation in the predicted trajectory leads to a collision.

Figure 3: Visual and action predictions evolve at different rates during denoising. Left: driving and manipulation examples show that visual structure becomes established earlier, while the estimated ego and gripper trajectories continue to change. Columns indicate denoising steps and noise levels. Right: schematic schedules illustrate the mismatch under shared noising. Our proposed warp instead assigns higher video noise at a given action noise level.

Figure 4: The architecture of CtrlWAM . A shared diffusion transformer processes video, ego-action, and optional agent-action tokens, conditioned on observed history. Future tokens are denoised as prediction targets or supplied clean as conditions, enabling joint prediction, action-conditioned video generation, and video-conditioned action prediction. The additional agent streams support prediction and control of surrounding agents.

Figure 5: Counterfactual ego control with CtrlWAM . Rows show the generation under the factual ego, and stop, accelerate commands. Columns show corresponding times from 1.8 to 6.4 s; the small plots at right show the ego trajectories associated with each row.

Figure 6: Ego-command counterfactuals on 100 clips. Each level is averaged over four commands (brake to stop, accelerate, nudge left, nudge right). Warped schedules (

\varsigma

= 2, 3, 5) follow commands more closely than the shared schedule (

\varsigma

= 1) at every level, with the gap widening as perturbations grow.

Table 2: Counterfactual agent-command following. Scene history, the ego command, and sampling noise are held fixed while a selected agent’s future trajectory changes. Gemini 3.8 Flash evaluates agreement between the generated video and the supplied agent command; scores are reported on a percentage scale.

Category	Cosmos 3 Zero-shot	Cosmos 3 SFT	GT-Src	CtrlWAM	Oracle
Visual Quality	0.447	0.560	0.563	0.576	0.598
Motion Quality	0.247	0.316	0.306	0.327	0.348
Content Consistency	0.413	0.650	0.620	0.665	0.662
Physics Adherence	0.258	0.456	0.480	0.540	0.782
3D Accuracy	0.812	0.884	0.899	0.921	0.956
Controllability	0.635	0.774	0.788	0.819	0.869

Table 3: WorldArena Track-1 summary ( Shang et al., 2026 ) on RoboTwin manipulation (

1{,}000

episodes). Rows report six category averages and EWMScore, which is

100\times

the mean of the 15 individual metrics. Oracle evaluates ground-truth videos and is excluded from rankings.

Figure 8: Qualitative RoboTwin manipulation comparison. Rows show the recorded video and generations from Cosmos-SFT, the GT-Src control, and CtrlWAM at matching frame indices. The trajectory-accuracy annotations apply to this episode, for which CtrlWAM more closely reproduces the recorded bowl motion and final arrangement.

Table 10

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Additional counterfactual ego control on a highway (top) and at an urban intersection (bottom). Each panel compares the recording with generations under factual, stop, accelerate, nudge-left, and nudge-right commands at matching times over 0.0–6.4 s. Plots at right show the corresponding ego trajectories.

Explore similar work

Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imagination
Jun 8, 2026Jiajun Li, Tiecheng Guo, Yifan Ye +9Efficient World-Action Model Faster-Wam
1The University of Hong Kong · 2Peking University · 3Muka Robotics +2
What Makes World Action Models Generalize? An Empirical Study of Test-Time Future Modeling
Sep 28, 2026Renping Zhou, Zanlin Ni, Zihao Fan +8Latent World Models World Models
Leap Lab, Tsinghua University · University of Science and Technology of China · Beijing Institute of Technology
ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?
Jun 17, 2026Yuyang Zhang, Wenyao Zhang, Zekun Qi +7World Models Video Generation
Shanghai Jiao Tong University · Tsinghua University · Tencent Robotics X +1

Jun 8, 2026, cs.ROJ/K move · Enter open · S save

Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imagination

Jiajun Li, Tiecheng Guo, Yifan Ye, Rongyu Zhang+8

1The University of Hong Kong · 2Peking University · 3Muka Robotics+2

Efficient World-Action Model Faster-Wam Future Video Prediction World Models Inference Latency Real-World Manipulation Tasks

Open paper AskPDF arXiv

World-Action Models (WAMs) have emerged as a promising paradigm for embodied control by coupling future visual prediction with action generation. However, most existing WAMs rely on photorealistic future prediction, which incurs high inference latency and makes real-time robot deployment difficult. This motivates a more efficient WAM design that preserves the control benefits of future visual prediction while reducing its inference cost. We introduce Efficient-WAM, a World-Action Model that reduces the cost of future imagination while preserving its control benefit. Efficient-WAM improves inference efficiency via a compact video expert transferred from WAN-2.2-5B, token-sparse video latents, and asymmetric video-action denoising that allocates fewer sampling steps to video than to actions. Instead of optimizing the future branch for visual fidelity, Efficient-WAM treats future video prediction as a compact guidance signal for action generation. Comprehensive experiments on RoboTwin 2.0 and real-world manipulation tasks show that Efficient-WAM maintains strong action performance despite visibly coarse future predictions. While maintaining competitive control capabilities, our 1B-parameter model can reduce per-chunk latency to around 100 ms during physical deployment, achieving a 30x speedup over existing WAMs.

Abstract

Explore similar work

Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imagination

What Makes World Action Models Generalize? An Empirical Study of Test-Time Future Modeling

ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?