Explicitly capturing demonstrator multi-modality for distillation into diffusion policies
Authors: Shao Yuan Chew Chia, Rebecca H. Jiang, Abhinav Agarwal, Andreas Hofmann, Leslie Pack Kaelbling, Russ Tedrake
Organizations: Computer Science and Artificial Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology, Cambridge, MA, USA · Pickle Robot Company, Charlestown, MA, USA
The diversity of a distilled diffusion policy's outputs matters beyond how it impacts the policy's success rate. In particular, downstream applications like test-time search and policy improvement require exploration. Thus, we want to distill multi-modal demonstrators into diffusion policies which reproduce their conditional action distributions. Standard data collection for behavior cloning applies a sliding window to demonstration episodes, which pairs unique observation histories with actions. These datasets can be fit by a policy that is either multi-modal, or unimodal with high Lipschitz constant to capture a diversity of actions seen at nearby observation samples. Prior work finds that diffusion policies trained on sparse data learn the latter, memorizing one action per observation. We adapt vine sampling to behavior cloning data collection to resolve this ambiguity: at some states the demonstrator visits, query it k times from exactly the same observation. On a synthetic task, we study how the number of actions per observation, k, affects the learned policy, and find that larger k yields a policy that better matches the demonstrator's conditional distribution, including its multi-modality and diversity. On a 2D truck unloading task, vine sampling likewise leads to a policy closer to the demonstrator and substantially improves both best-of-N and expert iteration performance.
Figures & tables
Figure 1: Actions from the truck unloading demonstrator of sec. C.1 (top), the vine policy (middle), and the sliding window policy (bottom), queried at the same state of a held-out problem instance. Each panel shows the observation image with future arm poses overlaid; poses with suction on are shown in pink. Of 10 samples each, the vine policy’s reach three of the four grasps, while the sliding window policy’s all approach the same grasp.
Figure 2: The choices that a demonstrator has the capacity to take branch along an episode. Sliding window datasets record one action per observation history, while vine datasets record several counterfactual actions at a random subset of observation histories. Gray actions are recorded in neither dataset.
Figure 3: (a) Coordinate y of the synthetic task at two observations, with k=5 actions sampled at each; arrows run from each observation to its four valid modes. (b) Visualization of synthetic datasets with fixed total dataset size n⋅k=1000 , but varied numbers of observations n , and actions per observations k ; gray diagonals show the valid modes of the task. At fixed dataset size, more actions per condition comes at the cost of sparsity in z .
Figure 4: Policies trained on the three datasets of fig. 3 , queried at their training observations. (a) Each policy’s draws at a representative training observation. Dashed segments and gray bands are πE and its support. Short black lines mark the training actions at each observation. (b)–(g) The metrics of sec. 4.2 , averaged over training observations; (c) shows the terms of ( 1 ) on the primary coordinate y . Reference lines are fresh draws from the demonstrator (black dashed, H0gen ) and resampled draws from the training actions (colored dashed, H0mem ).
Figure 5: Truck unloading experiment. (a), (b) Policy error W1(πθ,πE) and SD ratio per method at held-out conditions. (c), (d) Success rate and throughput, in best-of- N search; N=10 for solid bars except more search where N=100 . (e), (f) The same two metrics over expert iteration for the sliding window , SIME , and vine methods. Plotted values are means and error bars are 95% intervals.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Implementation
LeRobot DiffusionPolicy
Architecture
1D convolutional U-Net, one level of width 256
Parameters
3.1M
Diffusion steps
100 for training and sampling (DDPM)
Noise schedule
cosine
Prediction target
ϵ
Optimizer
AdamW, learning rate 10−4 , weight decay 10−6
Appendix
Table 1: Architecture and training recipe of the twelve synthetic policies.
Figure 6: The policies of fig. 4 , queried at held-out conditions at distance d from the nearest training observation, in units of the mean distance between two training observations. (a), (b) Each policy’s draws at a representative held-out condition at d=0.5 and d=1.0 . Dashed segments and gray bands are πE and its support. Short black lines mark the nearest training observation’s actions, shifted into the query’s residual frame. (c)–(h) The metrics of fig. 4 , averaged within bins of d ; diamonds at d=0 are the values at training observations. Reference lines are fresh draws from the demonstrator (black dashed, H0gen ).
Figure 7: The metrics of fig. 4 against the total dataset size n⋅k , one curve per k , averaged over training observations. Reference lines are fresh draws from the demonstrator (black dashed, H0gen ).
Architecture
CNN-based U-Net of Chi et al. [10] , channels (256,512,1024)
Image encoder
ResNet-18 trained from scratch, GroupNorm, spatial softmax
Figure 8: The SIME baseline’s noise sweep on the sliding window policy, against the noise scale σ ; σ=0 is the unmodified policy. (a) Policy error and (b) SD ratio at the states of fig. 5 , for the 100,000-step checkpoint (dark) and the 50,000-step one (light). (c) Success rate and (d) throughput on the 600 test instances at the 100,000-step checkpoint, under best-of-10 (filled) and for the raw policy (open). Reference lines are two independent sets of expert re-queries (black dashed) and the vine policy at 100,000 steps, under best-of-10 in (c) and (d) (green). The circled point is σ=4 , the value used in fig. 5 . Means, with 95% intervals as in fig. 5 .
Figure 9: The sliding window and vine policies against training step. (a) Policy error and (b) SD ratio at the states of fig. 5 . (c) Success rate under best-of-10 and (d) of the raw policy, on the 600 test instances. Reference lines in (a) and (b) are two independent sets of expert re-queries (black dashed).
Behavioral cloning becomes challenging when the same observation admits several valid actions. We study how generative behavioral-cloning policies represent such multimodal expert behavior and identify different bottlenecks across model parameterizations. For latent-variable policies, preserving demonstrated modes requires action-conditioned information in the latent representation. Excessive posterior-prior regularization can suppress this information and prevent the policy from distinguishing demonstrated modes. Weaker or aggregate regularization can preserve mode information, but shifts the challenge to ensuring that the deployment-time prior covers the relevant latent regions. For action-space generative policies, multimodality is constrained by the smoothness of the base-to-action transport: a map with a small Lipschitz constant cannot assign substantial probability to many well-separated modes. Covering many modes therefore requires either sharp transitions in base space or off-support bridge regions in action space. Experiments on synthetic multimodal navigation and a physical-robot bimodal manipulation task support these mechanisms. In contrast, our analysis reveals limited conditional multimodality in standard robotic simulation benchmarks, where deterministic regression remains competitive.
Lorenzo Mazza, Massimiliano Datres, Ariel Rodriguez +3
NCT/UCC Dresden, UKDD Dresden, TU Dresden, DKFZ Heidelberg · Ludwig-Maximilians-Universität München, Munich Center for Machine Learning (MCML) · BMFTR Research Hub 6G-Life +4
When diffusion policies were first introduced, they were expected to recover multi-modal action distributions. However, we find this expectation does not always hold, as diffusion policies often collapse to a single modality even when we guarantee the balance of dataset modalities and exact within-batch symmetry. Our analysis indicates that independent action-noise pairing contributes to this failure by increasing mixing and crossing among diffusion paths, which can produce averaged denoising responses and suppress modality-specific behavior. This issue is especially severe in robot planning, where action spaces are dense and low-dimensional, significantly increasing such mixing and crossing. To alleviate this problem, we propose Immiscible Diffusion Policy, a label-free training-time add-on to diffusion policy that uses action-noise assignment to preserve relatively distinct noise-to-action routes without modifying the policy architecture or inference procedure. Across five simulated and two real-world humanoid manipulation tasks spanning state, RGB, and point-cloud observations, our method significantly improves the policy's preservation of action modalities while maintaining strong task performance. It increases the proportion of the non-dominant modality by 6.0x-14.6x across three two-modality tasks and recovers demonstrated modalities that are entirely absent from vanilla policy rollouts on both four-modality tasks. These results demonstrate that Immiscible Diffusion Policy provides a simple yet robust approach to preserving action multi-modality in general robot learning tasks.
Xiao Zhang, Yuxin Chen, Zhixuan Liang +5
University of California, Berkeley · Princeton University · The University of Texas at Austin
We address the problem of fine-tuning pre-trained generative policies with reinforcement learning (RL) while preserving the multimodality of their action distributions. Existing methods for RL fine-tuning of generative policies (e.g., diffusion policies) improve task performance but often collapse diverse behaviors into a single reward-maximizing mode. To mitigate this issue, we propose an unsupervised mode discovery framework that uncovers latent behavioral modes within generative policies. The discovered modes enable the use of mutual information as an intrinsic reward, regularizing RL fine-tuning to enhance task success while maintaining behavioral diversity. Experiments on robotic manipulation tasks demonstrate that our method consistently outperforms conventional fine-tuning approaches, achieving higher success rates and preserving richer multimodal action distributions.
Alberta Longhini, David Emukpere, Jean-Michel Renders +1
Department of Computer Science, Stanford University, CA, USA