DiWALI: Diffusion Policy Learning in World Models via Latent Action Inference from Human and Robot Play
Authors: Akshay L Chandra, Iman Nematollahi, Lingyue Zhao, Daria Kulikova, Angela P. Schoellig, Fabien Despinoy, Abhinav Valada
Organizations: Department of Computer Science, University of Freiburg, Germany · Technical University of Munich, Germany · ZUSE School ELIZA · Toyota Motor Europe
Learning robot manipulation is constrained by the availability of action-labeled robot data, for both imitation and world models. Humans can demonstrate skills in seconds, but their demonstrations lack the required robot actions. We introduce DiWALI, a framework that learns and improves diffusion policies within a world model trained on human and robot play. DiWALI infers latent actions from consecutive frames using an inverse dynamics model, enabling world model training without action labels, and grounds them in robot commands through a decoder trained separately on robot play. Skill policies are trained over latent actions from action-free human demonstrations and improved with reinforcement learning in imagination. On the CALVIN benchmark, DiWALI policies trained solely from human demonstrations substantially outperform baselines trained with robot demonstrations. Incorporating human play into the world model training also yields stronger policies after fine-tuning than using the same amount of robot play alone. Experiments on a physical robot show that skills learned primarily from human demonstrations are further improved in imagination with a world model trained on human and robot play. Together, these results demonstrate that human experience can support both manipulation skill acquisition and subsequent policy improvement, reducing reliance on action-labeled robot data. Videos, dataset and code are available at https://diwali.cs.uni-freiburg.de.
Figures & tables
Fig. 1: From human demonstrations to robot skill: DiWALI trains one world model on robot play and action-free human play via latent actions inferred from consecutive frames. A diffusion policy imitates human demonstrations in this latent space, improves in imagination, and executes through a decoder trained on robot play.
Fig. 2: DiWALI framework. (a) A latent-action world model is trained on robot and human play, where an inverse dynamics model infers a latent action ut−1 from the embeddings of two consecutive observations, and the RSSM uses it in place of the robot command to advance its recurrent state, predict the prior z^t , and reconstruct the observation. No action labels are used. (b) The frozen world model encodes robot play into latent states zt and latent actions ut , from which an action decoder learns to predict the executed robot commands at . (c) Human demonstrations are encoded the same way, and a diffusion policy learns to denoise Gaussian noise into their latent actions conditioned on zt . (d) The policy is fine-tuned with PPO on imagined rollouts, in which the frozen world model advances the latent state under the policy’s latent actions, and a frozen reward classifier scores each state. Snowflakes mark frozen components, flames trainable ones.
Method
Policy
Demos
Success Rate (%)
Obs.
Action
Robot
Human
Drawer (2 tasks)
Slider (2 tasks)
Lightbulb (2 tasks)
LED (2 tasks)
Block (3 tasks)
Avg. (11 tasks)
Diffusion Policy (ViT)
RGB
EE- Δ
5
0
71.8 ± 5.8
38.8 ± 9.7
29.7 ± 9.7
28.0 ± 20.0
7.9 ± 12.2
35.2 ± 11.5
1
0
23.4 ± 23.3
7.9 ± 4.1
12.5 ± 10.9
7.7 ± 5.4
4.2 ± 8.4
11.1 ± 10.4
DiWA
WM
EE- Δ
5
0
39.0 ± 4.6
19.6 ± 9.8
32.6 ± 15.7
18.8 ± 10.7
7.8 ± 8.8
23.6 ± 9.9
1
0
1.1 ± 2.7
8.9 ± 6.4
2.6 ± 3.0
4.8 ± 7.4
3.5 ± 2.6
4.2 ± 4.4
CLAM
WM
Latent
5
0
9.1 ± 7.7
7.8 ± 7.4
6.1 ± 6.9
3.8 ± 1.2
7.3 ± 6.2
6.8 ± 5.9
TABLE I: Skill learning from few or no robot demonstrations on CALVIN. Success rates (%, mean ± std over three seeds) of behavior-cloned policies on 11 tasks in five groups. Obs. is the policy’s input, raw RGB or the world model’s latent state. Action is what it denoises, robot end-effector deltas or latent actions. Demos is the number of robot and human demonstrations per task. DiWALI outperforms every baseline on average, and with no robot demonstrations at all it surpasses baselines given five (59.6% vs. 35.2% average success). Best result per column in bold.
(a) Decoding latent actions
CLAM: joint decoder, robot play
6.8 ± 5.9
+ human play (CLAM-X)
9.5 ± 4.7
+ decoder trained separately
18.7 ± 9.8
+ decoder conditioned on latent state (DiWALI)
61.6 ± 11.1
(b) Auxiliary supervision for latent actions
None (DiWALI)
61.6 ± 11.1
TABLE II: Ablations. Success (%, mean ± std, three seeds) on 11 CALVIN tasks with 5 robot and 50 human demonstrations unless noted. (a) Decoder choices, added cumulatively to CLAM. (b) Auxiliary flow supervision. (c) Robot/human play mix at six hours.
Fig. 3: Robot and human play. Scene (top) and egocentric (bottom) views of simulated (a, b) and real-world (c, d) environments. (a) CALVIN and (b) CALVIN-H with a rendered human arm in place of the robot. (c) A real robot and (d) a real human acting in the real-world. In both domains, the human data has no action labels.
Method
Play
Demos
Before
After
Gain
R
H
DiWA
100% robot
50
0
54.7 ± 9.7
80.4 ± 4.9
+25.7
DiWA
50% robot
50
0
47.0 ± 10.9
69.5 ± 11.8
+23.0
DiWA
100% robot
5
0
28.2 ± 10.7
53.3 ± 11.7
+25.1
DiWA
50% robot
5
0
23.6 ± 9.9
50.4 ± 12.2
+26.8
DiWALI
50% robot, 50% human
5
0
48.1 ± 22.5
67.4 ± 13.7
+19.3
TABLE III: Policy improvement in imagination. Success rate (%, mean ± std over three seeds) across 11 tasks before and after PPO fine-tuning in the world model. Play denotes the world model’s training data. Demos are the robot (R) and human (H) demonstrations per task used to train the policy and reward classifier. Red marks higher-cost robot data, while green marks lower-cost or human data. DiWALI outperforms DiWA trained on the same robot play with ten times as many robot demonstrations.
Fig. 4: Real-world results. (a) The manipulation tasks used for evaluation. (b) Success before and after fine-tuning in imagination, averaged over 20 rollouts and three seeds. DiWALI, trained on human and robot play, improves skills learned primarily from human demonstrations without any physical interaction.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. A.1: The world model rolls out futures from latent actions for the CALVIN robot (top) and the CALVIN-H human arm (bottom), matching ground-truth simulator rollouts (left) with imagined rollouts (right).
Fig. A.2: The world model rolls out futures from latent actions for the real robot (top) and the real human (bottom), matching ground-truth rollouts (left) with imagined rollouts (right).
Parameter
Symbol
Value
Image resolution
—
64×64
Batch size
—
50
Sequence length (model steps)
—
50
Control steps per model step
H
5
Deterministic state dimensions
—
1024
Stochastic state (variables × classes)
—
32×32
Appendix
TABLE A.1: World model hyperparameters.
Parameter
Value
Input
latent state ⊕ latent action
Input dimensions
2064
Output dimensions
5×7
MLP dimensions
[1024, 512, 512, 256, 256]
Batch size
50
Optimizer
AdamW ( β1=0.9 , β2=0.999 )
Appendix
TABLE A.2: Action decoder hyperparameters.
Parameter
Symbol
Value
Observation horizon
—
1
Observation dimensions
—
2048
Action dimensions (latent)
d
16
Denoising steps
K
20
MLP dimensions
—
[512, 512, 512]
Training epochs
—
5000
Appendix
TABLE A.3: Diffusion policy hyperparameters, shared across all tasks and methods.
Parameter
Symbol
Value
Denoising steps
K
20
Fine-tuned denoising steps
Kft
10
Control steps per latent action
H
5
Parallel imagined rollouts
N
50
World-model steps (CALVIN / real)
—
∼ 1M / 216k
Actor / critic learning rate
—
10−5 / 10−3
Appendix
TABLE A.4: Fine-tuning hyperparameters, shared across tasks and with DiWA unless noted.
Fig. A.3: Auxiliary flow predictions decoded from latent actions on the static camera for robot play (left pairs) and human play (right pairs). Each pair shows, at different environment states, the observation and the flow prediction: (a) dense optical flow in standard color coding and (b) predicted point displacements overlaid on the observation.
Fig. A.4: (a) Real-world setup showing the Franka Emika Panda, VR teleoperation interface, camera placements, the 3D-printed wrist mount used for human play (b) Example static and egocentric observations from robot play (top) and human play (bottom).
This paper presents the World-Action Model (WAM), an action-regularized world model that jointly reasons over future visual observations and the actions that drive state transitions. Unlike conventional world models trained solely via image prediction, WAM incorporates an inverse dynamics objective into DreamerV2 that predicts actions from latent state transitions, encouraging the learned representations to capture action-relevant structure critical for downstream control. We evaluate WAM on enhancing policy learning across eight manipulation tasks from the CALVIN benchmark. We first pretrain a diffusion policy via behavioral cloning on world model latents, then refine it with model-based PPO inside the frozen world model. Without modifying the policy architecture or training procedure, WAM improves average behavioral cloning success from 59.4% to 71.2% over DreamerV2 and DiWA baselines. After PPO fine-tuning, WAM achieves 92.8% average success versus 79.8% for the baseline, with two tasks reaching 100%, using 8.7x fewer training steps.
Yuci Han, Alper Yilmaz
Photogrammetry and Computer Vision Lab, The Ohio State University, Columbus, OH 43210, USA.
Human videos provide rich information about object manipulation at a scale difficult to reproduce with robots, but often lack action annotations that can directly supervise robot policies. Realizing this potential requires distinguishing interaction-relevant changes from static appearance and connecting the learned interaction knowledge to executable robot actions. We present WALA, a framework named for World- and Action-supervised Latent Actions. Its semantic-geometric latent action model (LAM) learns latent actions by encoding and predicting semantic and geometric changes, emphasizing interactions over static appearance while retaining spatial detail. During robot policy training, the LAM encoder and decoder provide guidance through distillation and visual prediction, respectively. Together with robot action supervision, these signals shape policy-generated latent actions to support world prediction and capture the information needed to directly generate executable robot actions. Because LAM supervision requires no action labels, WALA enables co-training on action-labeled robot demonstrations and task-relevant action-free videos. In simulation experiments, our method achieves 92.33% average success on RoboTwin and 75.2% on RoboCasa-GR1-Tabletop, exceeding the previous state of the art on RoboCasa by 5.0 percentage points. WALA also shows strong real-robot performance, further enhanced by task-relevant action-free human videos that improve data efficiency and support target-task zero-shot and few-shot transfer.
Post-training is essential for turning pretrained generalist robot policies into reliable task-specific controllers, but existing human-in-the-loop pipelines remain tied to physical execution: each correction requires robot time, scene setup, resets, and operator supervision in the real world. Meanwhile, action-conditioned world models have been studied mainly for imagination, synthetic data generation, and policy evaluation. We propose Human-in-the-World-Model (Hi-WM), a post-training framework that uses a learned world model as a reusable corrective substrate for failure-targeted policy improvement. A policy is first rolled out in closed loop inside the world model; when the rollout becomes incorrect or failure-prone, a human intervenes directly in the model to provide short corrective actions. Hi-WM caches intermediate states and supports rollback and branching, allowing a single failure state to be reused for multiple corrective continuations and yielding dense supervision around behaviors that the base policy handles poorly. The resulting corrective trajectories are then added back to the training set for post-training. We evaluate Hi-WM on three real-world manipulation tasks spanning both rigid and deformable object interaction, and on two policy backbones. Hi-WM improves real-world success by 37.9 points on average over the base policy and by 19.0 points over a world-model closed-loop baseline, while world-model evaluation correlates strongly with real-world performance (r = 0.953). These results suggest that world models can serve not only as generators or evaluators, but also as effective corrective substrates for scalable robot post-training.
Yaxuan Li, Zhongyi Zhou, Yefei Chen +5
Current Robotics · Tsinghua University · Peking University +1