Learning to Be Different: Component Specialization for Compositional Diffusion Policies

Alec Laprevotte; Haonan Chen; Xiaoshen Han; Yilun Du

cs.ROOct 19, 2026

Learning to Be Different: Component Specialization for Compositional Diffusion Policies

Authors: Alec Laprevotte, Haonan Chen, Xiaoshen Han, Yilun Du

Organizations: Harvard University

Abstract

Diffusion policies with linearly composed scores have emerged as a promising approach to multitask robotic learning by combining separate factors to enable modular, task-wise adaptation, compositional generalization, and interpretable functional decomposition. However, current training objectives do not encourage factor specialization, often resulting in uninterpretable denoisers whose composed prediction is an average over identical copies, undermining their architectural premise. To address this, we propose a drop-in score-decorrelation auxiliary loss, motivated by projective composition theory. Our approach encourages specialization by penalizing overlap directly between component noise predictions, with little architectural change or computational overhead. We demonstrate across multiple benchmarks that our decorrelation loss consistently reduces measurable double-counting between components, improves task decomposition, and produces behaviorally meaningful specialization without compromising multitask performance. Website: https://al3cl1.github.io/robotics-score-decorrelation.

Figures & tables

	RLBench
Policy	Close Box	Close Drawer	Close Fridge	Close Microwave	Toilet Seat Down	Take Umbrella	Reach Target	Avg.
DP	$9.60_{\pm 1.60}$	$99.1_{\pm 0.80}$	$71.2_{\pm 4.12}$	$51.2_{\pm 2.33}$	$52.8_{\pm 4.2}$	$13.6_{\pm 0.90}$	$16.0_{\pm 2.82}$	$44.8$
FDP	36.8 ±1.50	97.6 ±1.60	71.2 ±1.94	80.0 ±1.79	68.0 ±2.51	15.2 ±2.27	24.0 ±2.50	56.1
FDP + $\lambda_{0.5}$	49.6 ±3.25	97.6 ±1.60	81.6 ±3.15	67.2 ±4.07	66.4 ±4.45	10.4 ±3.11	14.4 ±3.71	55.3
FDP + $\lambda_{1.25}$	35.2 ±1.50	99.2 ±0.80	73.6 ±1.92	75.2 ±2.37	56.0 ±1.53	17.6 ±1.83	17.6 ±1.59	53.5
FDP + $\lambda_{2.5}$	39.2 ±2.87	98.4 ±1.60	63.2 ±2.96	67.2 ±1.66	62.4 ±2.30	16.8 ±1.20	23.2 ±2.37	52.9

Table 2: Task success rates. Mean rates across RLBench and RoboCasa atomics, averaged over 10 seeds with 220 episodes each.

\lambda

values are

\times 10^{-3}

. Decorrelation stays close to FDP on MT8 and improves RoboCasa.

Table 3

	Composite		Subgoal
$\lambda$ ( $\times 10^{-3}$ )	$K{=}4$	$K{=}8$	$K{=}4$	$K{=}8$
0	$15.3_{\pm 2.5}$	$21.7_{\pm 1.7}$	$24.4_{\pm 0.4}$	$35.3_{\pm 1.5}$
0.3	$\mathbf{23.1}_{\pm 2.4}$	$23.9_{\pm 1.0}$	$\mathbf{29.3}_{\pm 1.0}$	$35.3_{\pm 1.0}$
1	$15.8_{\pm 1.3}$	$29.2_{\pm 0.5}$	$26.5_{\pm 0.7}$	$37.2_{\pm 1.2}$
3	$10.8_{\pm 1.7}$	$\mathbf{35.3}_{\pm 2.0}$	$20.1_{\pm 0.7}$	$\mathbf{46.7}_{\pm 2.5}$
10	$16.4_{\pm 1.0}$	$20.3_{\pm 0.6}$	$24.0_{\pm 2.4}$	$35.3_{\pm 0.8}$

Table 5: Composite and subgoal success on LIBERO-10. Composite requires completing both subgoals, while subgoal gives credit for each: the best

\lambda

improves both.

Panel of Figure 2 — Figure 2: Real-world cup flipping. Agent-view (top) and wrist-view (bottom) frames, left to right, on an ARX R5 arm. Over 25 trials, FDP with our loss succeeds 19 times, versus 15 for FDP and 10 for DP.

Figure 5: Per-factor magnitude across action dimensions. Mean weighted absolute noise prediction of each factor on each action dimension. Without our loss (left), the four factors look identical; with it (right), factor 1 takes the base and factor 2 the gripper, with factors 0 and 3 sharing the arm.

Figure 6: Action attribution across horizon and denoising steps. With our loss, factor 1 owns the base and factor 2 the gripper at every step.

Figure 7: Rollout snippets of each factor’s action channel , $K{=}4$ with our loss on RoboCasa. Factor 0 makes fine arm moves, 1 drives the base, 2 grasps, and 3 assists large arm moves; time runs left to right.

Diffusion Policy

Diffusion policies generate multimodal action sequences by iteratively denoising noise, enabling imitation, planning, and control in robotics.

See all 202 papers

Factored Diffusion Policies:Compositionally Generalized Robot Control with a Single Score Network
May 21, 2026Sayan Mitra, Ege Yuceel, Noah Giles +1Robotic Control Autonomous Drone Racing
University of Illinois Urbana-Champaign
Immiscible Diffusion Policy: Preserving Multimodal Robot Actions through Label-Free Noise Assignment
Oct 7, 2026Xiao Zhang, Yuxin Chen, Zhixuan Liang +5Diffusion Policy Mode Collapse
University of California, Berkeley · Princeton University · The University of Texas at Austin
Semantically Structured Mixture-of-Experts for Compositional Robotic Manipulation
May 22, 2026Chengyu Deng, Guanqi Chen, Yizhou Chen +4Robot Policy Learning Robot Skill Learning
The University of Hong Kong · Southern University of Science and Technology

May 21, 2026J/K move · Enter open · S save

Factored Diffusion Policies:Compositionally Generalized Robot Control with a Single Score Network

Sayan Mitra, Ege Yuceel, Noah Giles, Abhishek Pai

University of Illinois Urbana-Champaign

Robotic Control Autonomous Drone Racing Diffusion Policy Data-Efficient Robot Learning Robot Policy Generalization Policy Learning+1

Open paper AskPDF arXiv

Robotic tasks are typically specified by a tuple of factors, such as the object to be grasped, the obstacles to be avoided, the color of the target, and so on. Collecting expert demonstrations for every combination of factor values grows combinatorially. We present factored diffusion policies: a single shared diffusion network trained with per-factor null-token dropout, whose score decomposes additively across factors at inference. Under approximate conditional independence between factors given the action-observation pair, this composition approximates the true joint score with a bounded uniform error, reducing the training-task budget from a product of factor cardinalities to a sum. A trajectory-tube certificate chains this score-level bound through the reverse-time sampling ODE and a contracting tracking controller into a closed-loop state-trajectory tube whose radius factors into an ODE-sensitivity constant and a per-factor score-error budget. Unlike compositional-diffusion methods for control that combine separately trained networks, we use one shared network. Drone racing experiments confirm both the generalization bound and the certificate. On state-based multi-gate racing, the factored policy passes 90% of held-out gates -- matching an oracle -- while a K-network composition baseline collapses to 3%; on vision-based single-gate traversal, it transfers zero-shot to an unseen venue with +11.7pp success-rate gain and 2.4X crash-rate reduction.

Abstract

Diffusion Policy

Explore similar work

Factored Diffusion Policies:Compositionally Generalized Robot Control with a Single Score Network

Immiscible Diffusion Policy: Preserving Multimodal Robot Actions through Label-Free Noise Assignment

Semantically Structured Mixture-of-Experts for Compositional Robotic Manipulation