DiWALI: Diffusion Policy Learning in World Models via Latent Action Inference from Human and Robot Play
Authors: Akshay L Chandra, Iman Nematollahi, Lingyue Zhao, Daria Kulikova, Angela P. Schoellig, Fabien Despinoy, Abhinav Valada
Organizations: Department of Computer Science, University of Freiburg, Germany · Technical University of Munich, Germany · ZUSE School ELIZA · Toyota Motor Europe
Learning robot manipulation is constrained by the availability of action-labeled robot data, for both imitation and world models. Humans can demonstrate skills in seconds, but their demonstrations lack the required robot actions. We introduce DiWALI, a framework that learns and improves diffusion policies within a world model trained on human and robot play. DiWALI infers latent actions from consecutive frames using an inverse dynamics model, enabling world model training without action labels, and grounds them in robot commands through a decoder trained separately on robot play. Skill policies are trained over latent actions from action-free human demonstrations and improved with reinforcement learning in imagination. On the CALVIN benchmark, DiWALI policies trained solely from human demonstrations substantially outperform baselines trained with robot demonstrations. Incorporating human play into the world model training also yields stronger policies after fine-tuning than using the same amount of robot play alone. Experiments on a physical robot show that skills learned primarily from human demonstrations are further improved in imagination with a world model trained on human and robot play. Together, these results demonstrate that human experience can support both manipulation skill acquisition and subsequent policy improvement, reducing reliance on action-labeled robot data. Videos, dataset and code are available at https://diwali.cs.uni-freiburg.de.
Figures & tables
Fig. 1: From human demonstrations to robot skill: DiWALI trains one world model on robot play and action-free human play via latent actions inferred from consecutive frames. A diffusion policy imitates human demonstrations in this latent space, improves in imagination, and executes through a decoder trained on robot play.
Fig. 2: DiWALI framework. (a) A latent-action world model is trained on robot and human play, where an inverse dynamics model infers a latent action ut−1 from the embeddings of two consecutive observations, and the RSSM uses it in place of the robot command to advance its recurrent state, predict the prior z^t , and reconstruct the observation. No action labels are used. (b) The frozen world model encodes robot play into latent states zt and latent actions ut , from which an action decoder learns to predict the executed robot commands at . (c) Human demonstrations are encoded the same way, and a diffusion policy learns to denoise Gaussian noise into their latent actions conditioned on zt . (d) The policy is fine-tuned with PPO on imagined rollouts, in which the frozen world model advances the latent state under the policy’s latent actions, and a frozen reward classifier scores each state. Snowflakes mark frozen components, flames trainable ones.
Method
Policy
Demos
Success Rate (%)
Obs.
Action
Robot
Human
Drawer (2 tasks)
Slider (2 tasks)
Lightbulb (2 tasks)
LED (2 tasks)
Block (3 tasks)
Avg. (11 tasks)
Diffusion Policy (ViT)
RGB
EE- Δ
5
0
71.8 ± 5.8
38.8 ± 9.7
29.7 ± 9.7
28.0 ± 20.0
7.9 ± 12.2
35.2 ± 11.5
1
0
23.4 ± 23.3
7.9 ± 4.1
12.5 ± 10.9
7.7 ± 5.4
4.2 ± 8.4
11.1 ± 10.4
DiWA
WM
EE- Δ
5
0
39.0 ± 4.6
19.6 ± 9.8
32.6 ± 15.7
18.8 ± 10.7
7.8 ± 8.8
23.6 ± 9.9
1
0
1.1 ± 2.7
8.9 ± 6.4
2.6 ± 3.0
4.8 ± 7.4
3.5 ± 2.6
4.2 ± 4.4
CLAM
WM
Latent
5
0
9.1 ± 7.7
7.8 ± 7.4
6.1 ± 6.9
3.8 ± 1.2
7.3 ± 6.2
6.8 ± 5.9
TABLE I: Skill learning from few or no robot demonstrations on CALVIN. Success rates (%, mean ± std over three seeds) of behavior-cloned policies on 11 tasks in five groups. Obs. is the policy’s input, raw RGB or the world model’s latent state. Action is what it denoises, robot end-effector deltas or latent actions. Demos is the number of robot and human demonstrations per task. DiWALI outperforms every baseline on average, and with no robot demonstrations at all it surpasses baselines given five (59.6% vs. 35.2% average success). Best result per column in bold.
(a) Decoding latent actions
CLAM: joint decoder, robot play
6.8 ± 5.9
+ human play (CLAM-X)
9.5 ± 4.7
+ decoder trained separately
18.7 ± 9.8
+ decoder conditioned on latent state (DiWALI)
61.6 ± 11.1
(b) Auxiliary supervision for latent actions
None (DiWALI)
61.6 ± 11.1
TABLE II: Ablations. Success (%, mean ± std, three seeds) on 11 CALVIN tasks with 5 robot and 50 human demonstrations unless noted. (a) Decoder choices, added cumulatively to CLAM. (b) Auxiliary flow supervision. (c) Robot/human play mix at six hours.
Fig. 3: Robot and human play. Scene (top) and egocentric (bottom) views of simulated (a, b) and real-world (c, d) environments. (a) CALVIN and (b) CALVIN-H with a rendered human arm in place of the robot. (c) A real robot and (d) a real human acting in the real-world. In both domains, the human data has no action labels.
Method
Play
Demos
Before
After
Gain
R
H
DiWA
100% robot
50
0
54.7 ± 9.7
80.4 ± 4.9
+25.7
DiWA
50% robot
50
0
47.0 ± 10.9
69.5 ± 11.8
+23.0
DiWA
100% robot
5
0
28.2 ± 10.7
53.3 ± 11.7
+25.1
DiWA
50% robot
5
0
23.6 ± 9.9
50.4 ± 12.2
+26.8
DiWALI
50% robot, 50% human
5
0
48.1 ± 22.5
67.4 ± 13.7
+19.3
TABLE III: Policy improvement in imagination. Success rate (%, mean ± std over three seeds) across 11 tasks before and after PPO fine-tuning in the world model. Play denotes the world model’s training data. Demos are the robot (R) and human (H) demonstrations per task used to train the policy and reward classifier. Red marks higher-cost robot data, while green marks lower-cost or human data. DiWALI outperforms DiWA trained on the same robot play with ten times as many robot demonstrations.
Fig. 4: Real-world results. (a) The manipulation tasks used for evaluation. (b) Success before and after fine-tuning in imagination, averaged over 20 rollouts and three seeds. DiWALI, trained on human and robot play, improves skills learned primarily from human demonstrations without any physical interaction.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. A.1: The world model rolls out futures from latent actions for the CALVIN robot (top) and the CALVIN-H human arm (bottom), matching ground-truth simulator rollouts (left) with imagined rollouts (right).
Fig. A.2: The world model rolls out futures from latent actions for the real robot (top) and the real human (bottom), matching ground-truth rollouts (left) with imagined rollouts (right).
Parameter
Symbol
Value
Image resolution
—
64×64
Batch size
—
50
Sequence length (model steps)
—
50
Control steps per model step
H
5
Deterministic state dimensions
—
1024
Stochastic state (variables × classes)
—
32×32
Appendix
TABLE A.1: World model hyperparameters.
Parameter
Value
Input
latent state ⊕ latent action
Input dimensions
2064
Output dimensions
5×7
MLP dimensions
[1024, 512, 512, 256, 256]
Batch size
50
Optimizer
AdamW ( β1=0.9 , β2=0.999 )
Appendix
TABLE A.2: Action decoder hyperparameters.
Parameter
Symbol
Value
Observation horizon
—
1
Observation dimensions
—
2048
Action dimensions (latent)
d
16
Denoising steps
K
20
MLP dimensions
—
[512, 512, 512]
Training epochs
—
5000
Appendix
TABLE A.3: Diffusion policy hyperparameters, shared across all tasks and methods.
Parameter
Symbol
Value
Denoising steps
K
20
Fine-tuned denoising steps
Kft
10
Control steps per latent action
H
5
Parallel imagined rollouts
N
50
World-model steps (CALVIN / real)
—
∼ 1M / 216k
Actor / critic learning rate
—
10−5 / 10−3
Appendix
TABLE A.4: Fine-tuning hyperparameters, shared across tasks and with DiWA unless noted.
Fig. A.3: Auxiliary flow predictions decoded from latent actions on the static camera for robot play (left pairs) and human play (right pairs). Each pair shows, at different environment states, the observation and the flow prediction: (a) dense optical flow in standard color coding and (b) predicted point displacements overlaid on the observation.
Fig. A.4: (a) Real-world setup showing the Franka Emika Panda, VR teleoperation interface, camera placements, the 3D-printed wrist mount used for human play (b) Example static and egocentric observations from robot play (top) and human play (bottom).