De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control

Shaoyi Zheng; Zhongtao Jiang; Yue Peng; Wenbo Lu; Shunchi Zhang; Xi Wang; Shengji Wan

cs.ROOct 20, 2026

De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control

Authors: Shaoyi Zheng, Zhongtao Jiang, Yue Peng, Wenbo Lu, Shunchi Zhang, Xi Wang, Shengji Wan

Organizations: ByteDance. · New York University (NYU).

Abstract

Cascade world-action models (WAMs) use predicted visual futures to condition robot actions, but inherit costly iterative inference from diffusion-pretrained experts. We ask what a single-step predicted future must preserve to support accurate single-step action prediction. In the evaluated WAM, resampling video noise has little effect on downstream actions, yet naively truncating both streams to one step substantially reduces task success. Video-space supervision only partially closes this gap, and a local sensitivity analysis shows that vision prediction errors have highly unequal consequences for action accuracy. These findings motivate DirectWAM, which converts pretrained video and action experts into single-step predictors without stochastic sampling through action-sufficient distillation, while retaining the explicit future-to-action cascade. After learning a one-step action model, we train the future predictor by decoding actions from its own predicted futures and backpropagating demonstration action error through the expert, with an optional future-latent reconstruction loss. The resulting objective evaluates the student-generated future through its downstream control consequences rather than relying on representation matching alone. DirectWAM achieves 97.1% average success across four LIBERO suites and 85.4% on RoboTwin 2.0, outperforming Flash-WAM at the same one-evaluation-per-stream budget. This advantage has also extended to three real-world tasks that we designed. Compared with the full iterative teacher, it reduces inference latency by $28.0\times$ .

Figures & tables

Figure 1: Overview of DirectWAM. (a) Video randomness produces similar downstream actions. (b) Naive step truncation and visual supervision induce a noticeable performance gap. (c) DirectWAM is a deterministic one-step cascade WAM trained through action-sufficient distillation.

Benchmark	Contexts $\times$ samples	$\rho(D_{V},D_{A})$	Mean $D_{A}/D_{V}$	V $\to$ A var ratio
RoboTwin	$237\times 32$	0.69	0.025	$6.6\%\to 0.030\%$
LIBERO	$64\times 16$	0.80	0.138	$1.6\%\to 0.25\%$

Table 1: Three metrics measuring the effect of video variation on downstream actions. Same-context video-sampling variation is substantially attenuated by the action model.

Panel of Figure 2 — Figure 2: Closed-loop effects of video-side stochasticity. (a) RoboTwin success rates of the teacher, fixed video noise, and averaging four future predictions. (b) Eight rollouts per scene under different video-noise seeds. Successful trajectories remain closely aligned.

Panel of Figure 3 — Figure 3: Inference-step reduction and action-sensitive error recovery. (a) Success rates (%) across video/action step budgets. Performance degrades as the number of steps decreases. (b) The action-sensitive component holds only $0.63$ – $0.85\%$ of the discrepancy energy yet recovers $19.8$ – $34.0\%$ of the offline action-error improvement.

Method	Steps (Video / Action)	RoboTwin 2.0			LIBERO
Method	Steps (Video / Action)	Clean	Rand.	Average.	Long	Goal	Object	Spatial	Average.
Lingbot Baseline	Full / 50	91.2	90.2	90.7	98.2	93.0	97.2	98.0	96.6
Naive Step Reduction	2 / 1	76.2	71.5	73.9	97.0	90.8	96.8	96.6	95.3
	1 / 2	67.8	68.0	67.9	90.8	82.8	94.0	94.2	90.5
	1 / 1	57.3	52.4	54.9	88.6	77.8	87.8	91.2	86.4
Flash-WAM	1 / 1	82.1	74.2	78.2	95.8 †	96.0 †	92.6 †	96.0 †	95.1 †

Table 2: Main simulation results for cascade WAMs. Success rates (%) on RoboTwin 2.0 and LIBERO; higher is better. The full iterative LingBot teacher is italicized. Steps are video/action steps; Full denotes 20 video steps for LIBERO and 25 for RoboTwin, following the respective LingBot-VA evaluation configurations. RoboTwin results use 50 episodes per task across all 50 tasks. Bold denotes the better result between learned single-step methods (Flash-WAM and DirectWAM). The † denotes LIBERO results quoted from Flash-WAM because no corresponding checkpoint was released; these entries are therefore not treated as a strictly matched reproduction.

Panel of Figure 4 — Figure 4: Efficiency and qualitative comparison on RoboTwin. (a) Success–latency trade-off; labels show speedups over full LingBot-VA (5042 ms). (b) Six examples of predicted futures from the full teacher, the one-step teacher, Flash-WAM, and DirectWAM.

Method	Grasp Banana	Pouring Water	Place Teapot
LingBot-VA	95 (19/20)	45 (9/20)	60 (12/20)
DirectWAM	95 (19/20)	55 (11/20)	60 (12/20)
Flash-WAM	90 (18/20)	10 (2/20)	40 (8/20)

Table 3: Real-world task success rates. Each method is evaluated over 20 trials per task. Results are reported as success rates (%) with successful/total trials. Bold indicates the best result per task.

Figure 5: Real-world task trajectories. Six keyframes per task are ordered from left to right. Each keyframe pairs the external view (left) with the wrist view (right).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Contexts $\times$ samples	$\rho(D_{V},D_{A})$	Mean $D_{A}/D_{V}$	$R_{V}$	$R_{A}$	RMS attenuation
RoboTwin	$237\times 32$	0.69	0.025	6.6%	0.030%	$20\times$
LIBERO	$64\times 16$	0.80	0.138	1.6%	0.25%	$3\times$

Appendix

Table 4: Offline propagation of video-sampling variation. All actions within a benchmark are evaluated with the same frozen single-step model.

R_{V}

and

R_{A}

are fractions of total variance. The final column reports the separately recorded median RMS attenuation.

Model	Standard	Fixed 123	Fixed 456	Fixed 789	Fixed-noise mean	Avg-4
Iterative teacher	91.6	92.4	91.6	89.6	91.2	92.0
Single-step model	87.0	87.4	87.6	89.4	88.1	87.0

Appendix

Table 5: Video-sampling interventions on RoboTwin Clean. Success rates are percentages over 500 episodes per entry (50 tasks, 10 episodes per task). Fixed-noise mean is the arithmetic mean of the three seed-specific success rates.

Sample set	PCA-8	PCA-16
Same-context futures	$0/237$ (0.0%)	$1/237$ (0.4%)
Matched Gaussian controls	0.16%	0.0%
Two-context mixtures	97.4%	99.5%

Appendix

Table 6: Multimodality diagnostic and controls. Real-data entries give detected contexts over the 237-context audited set. Control entries report detection rates.

Figure 6: Stage-2 training dynamics. The action expert is frozen during the plotted phase. RoboTwin (left) uses $\lambda_{v}=0.01$ and shows both action loss and the weighted future-anchor loss. LIBERO-Long (right) uses $\lambda_{v}=0$ and shows only action loss. The vertical scale is logarithmic.

Prediction	GT-action error $\downarrow$	Recovery $\uparrow$
Vision-supervised $z^{\mathrm{V}}$	0.257	0%
Full replacement $z^{\mathrm{A}}$	0.216	100%
Selective replacement $z^{\mathrm{sens}}$	0.224	80%
Equal-norm random controls	–	1–2%

Appendix

Table 7: Selective replacement of learned prediction differences. Results use 64 RoboTwin contexts and the same frozen action model. The selected component accounts for 1.9% of the prediction-difference energy and recovers approximately 80% of the pooled offline action-error improvement. Dashes indicate that only the recovery summary is reported for the random controls.

Future representation	Transferred energy	Success rate (%) $\uparrow$
Vision-supervised $z^{V}$	0%	78.2
Selective replacement $z^{V}+P(z^{A}-z^{V})$	1.4%	87.3
DirectWAM $z^{A}$	100%	87.0

Appendix

Table 8: Closed-loop selective replacement of learned prediction differences. We keep the same downstream action model fixed and vary only the future representation supplied to it. The selective correction transfers only the locally action-sensitive component of the difference between the vision-supervised and DirectWAM futures.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	KID $\times 10^{3}$ $\downarrow$
Teacher, 25 steps	30.37	0.938	0.051	2.8	0.56
Teacher, 8 steps	30.16	0.936	0.054	3.5	0.93
Teacher, 4 steps	29.64	0.932	0.060	5.0	1.78
Teacher, 2 steps	28.57	0.920	0.075	8.1	3.74
Teacher, 1 step	25.80	0.885	0.123	17.4	10.24
Flash-WAM	26.62	0.928	0.077	10.5	4.11

Appendix

Table 9: Visual prediction quality. Teacher variants use guidance scale 5 with the indicated number of video denoising steps. Flash-WAM and DirectWAM use one step. Bold indicates the best value among the listed methods.

Efficient Inference for World Action Models

Efficient inference for world action models cuts the cost of predicting action-conditioned futures for responsive control and planning.

See all 80 papers

Vid2WAM: Distilling Video Diffusion Priors into World Action Models
Aug 9, 2026Chenhao Qiu, Ruixiang Wang, Runyi Zhao +7Cross-Modal Knowledge Distillation World Model Learning
Fudan University · The Chinese University of Hong Kong, Shenzhen · Shanghai Innovation Institute
Foresight Without Seeing: Latent Futures for World Action Models
Aug 12, 2026Jiakai Huang, Zhongbo Wu, Siyu Xu +5World Models for Robotics Visuomotor Policy Learning
Shanghai Jiao Tong University · ACE Robotics · Nanyang Technological University
Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imagination
Jun 8, 2026Jiajun Li, Tiecheng Guo, Yifan Ye +9World Models for Robotics Efficient Inference for World Action Models
1The University of Hong Kong · 2Peking University · 3Muka Robotics +2

Aug 9, 2026J/K move · Enter open · S save

Vid2WAM: Distilling Video Diffusion Priors into World Action Models

Chenhao Qiu, Ruixiang Wang, Runyi Zhao, Sixu Lin+6

Fudan University · The Chinese University of Hong Kong, Shenzhen · Shanghai Innovation Institute

Cross-Modal Knowledge Distillation World Model Learning Visuomotor Policy Learning World Action Models

Open paper AskPDF arXiv

World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.

Abstract

Efficient Inference for World Action Models

Explore similar work

Vid2WAM: Distilling Video Diffusion Priors into World Action Models

Foresight Without Seeing: Latent Futures for World Action Models

Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imagination