Diffusion policies with linearly composed scores have emerged as a promising approach to multitask robotic learning by combining separate factors to enable modular, task-wise adaptation, compositional generalization, and interpretable functional decomposition. However, current training objectives do not encourage factor specialization, often resulting in uninterpretable denoisers whose composed prediction is an average over identical copies, undermining their architectural premise. To address this, we propose a drop-in score-decorrelation auxiliary loss, motivated by projective composition theory. Our approach encourages specialization by penalizing overlap directly between component noise predictions, with little architectural change or computational overhead. We demonstrate across multiple benchmarks that our decorrelation loss consistently reduces measurable double-counting between components, improves task decomposition, and produces behaviorally meaningful specialization without compromising multitask performance. Website: https://al3cl1.github.io/robotics-score-decorrelation.
Figures & tables
Figure 1
RLBench
Policy
Close Box
Close Drawer
Close Fridge
Close Microwave
Toilet Seat Down
Take Umbrella
Reach Target
Avg.
DP
9.60±1.60
99.1±0.80
71.2±4.12
51.2±2.33
52.8±4.2
13.6±0.90
16.0±2.82
44.8
FDP
36.8 ±1.50
97.6 ±1.60
71.2 ±1.94
80.0 ±1.79
68.0 ±2.51
15.2 ±2.27
24.0 ±2.50
56.1
FDP + λ0.5
49.6 ±3.25
97.6 ±1.60
81.6 ±3.15
67.2 ±4.07
66.4 ±4.45
10.4 ±3.11
14.4 ±3.71
55.3
FDP + λ1.25
35.2 ±1.50
99.2 ±0.80
73.6 ±1.92
75.2 ±2.37
56.0 ±1.53
17.6 ±1.83
17.6 ±1.59
53.5
FDP + λ2.5
39.2 ±2.87
98.4 ±1.60
63.2 ±2.96
67.2 ±1.66
62.4 ±2.30
16.8 ±1.20
23.2 ±2.37
52.9
Table 2: Task success rates. Mean rates across RLBench and RoboCasa atomics, averaged over 10 seeds with 220 episodes each. λ values are ×10−3 . Decorrelation stays close to FDP on MT8 and improves RoboCasa.
Table 3
Composite
Subgoal
λ ( ×10−3 )
K=4
K=8
K=4
K=8
0
15.3±2.5
21.7±1.7
24.4±0.4
35.3±1.5
0.3
23.1±2.4
23.9±1.0
29.3±1.0
35.3±1.0
1
15.8±1.3
29.2±0.5
26.5±0.7
37.2±1.2
3
10.8±1.7
35.3±2.0
20.1±0.7
46.7±2.5
10
16.4±1.0
20.3±0.6
24.0±2.4
35.3±0.8
Table 5: Composite and subgoal success on LIBERO-10. Composite requires completing both subgoals, while subgoal gives credit for each: the best λ improves both.
Figure 2: Real-world cup flipping. Agent-view (top) and wrist-view (bottom) frames, left to right, on an ARX R5 arm. Over 25 trials, FDP with our loss succeeds 19 times, versus 15 for FDP and 10 for DP.
Figure 6Figure 7
Figure 5: Per-factor magnitude across action dimensions. Mean weighted absolute noise prediction of each factor on each action dimension. Without our loss (left), the four factors look identical; with it (right), factor 1 takes the base and factor 2 the gripper, with factors 0 and 3 sharing the arm.
Figure 6: Action attribution across horizon and denoising steps. With our loss, factor 1 owns the base and factor 2 the gripper at every step.
Figure 7: Rollout snippets of each factor’s action channel , K=4 with our loss on RoboCasa. Factor 0 makes fine arm moves, 1 drives the base, 2 grasps, and 3 assists large arm moves; time runs left to right.
Robotic tasks are typically specified by a tuple of factors, such as the object to be grasped, the obstacles to be avoided, the color of the target, and so on. Collecting expert demonstrations for every combination of factor values grows combinatorially. We present factored diffusion policies: a single shared diffusion network trained with per-factor null-token dropout, whose score decomposes additively across factors at inference. Under approximate conditional independence between factors given the action-observation pair, this composition approximates the true joint score with a bounded uniform error, reducing the training-task budget from a product of factor cardinalities to a sum. A trajectory-tube certificate chains this score-level bound through the reverse-time sampling ODE and a contracting tracking controller into a closed-loop state-trajectory tube whose radius factors into an ODE-sensitivity constant and a per-factor score-error budget. Unlike compositional-diffusion methods for control that combine separately trained networks, we use one shared network. Drone racing experiments confirm both the generalization bound and the certificate. On state-based multi-gate racing, the factored policy passes 90% of held-out gates -- matching an oracle -- while a K-network composition baseline collapses to 3%; on vision-based single-gate traversal, it transfers zero-shot to an unseen venue with +11.7pp success-rate gain and 2.4X crash-rate reduction.
When diffusion policies were first introduced, they were expected to recover multi-modal action distributions. However, we find this expectation does not always hold, as diffusion policies often collapse to a single modality even when we guarantee the balance of dataset modalities and exact within-batch symmetry. Our analysis indicates that independent action-noise pairing contributes to this failure by increasing mixing and crossing among diffusion paths, which can produce averaged denoising responses and suppress modality-specific behavior. This issue is especially severe in robot planning, where action spaces are dense and low-dimensional, significantly increasing such mixing and crossing. To alleviate this problem, we propose Immiscible Diffusion Policy, a label-free training-time add-on to diffusion policy that uses action-noise assignment to preserve relatively distinct noise-to-action routes without modifying the policy architecture or inference procedure. Across five simulated and two real-world humanoid manipulation tasks spanning state, RGB, and point-cloud observations, our method significantly improves the policy's preservation of action modalities while maintaining strong task performance. It increases the proportion of the non-dominant modality by 6.0x-14.6x across three two-modality tasks and recovers demonstrated modalities that are entirely absent from vanilla policy rollouts on both four-modality tasks. These results demonstrate that Immiscible Diffusion Policy provides a simple yet robust approach to preserving action multi-modality in general robot learning tasks.
Xiao Zhang, Yuxin Chen, Zhixuan Liang +5
University of California, Berkeley · Princeton University · The University of Texas at Austin
Diffusion-based policies have established a new standard for precise robotic manipulation but face a critical scalability bottleneck: high-performance models are computationally expensive, while lightweight alternatives often fail to generalize across diverse multi-task environments. Mixture-of-Experts (MoE) architectures offer a promising path to efficiency by activating only a subset of parameters. However, existing MoE routing mechanisms typically rely on low-level noise or latent statistics, ignoring the compositional nature of manipulation tasks. This can fragment reusable behaviors across experts, limiting interpretability and transferability. We introduce Semantically Structured Mixture-of-Experts Diffusion Policy (SMoDP) for compositional robotic manipulation, a framework that grounds expert specialization in semantic task structure. SMoDP leverages a lightweight, inference-time skill predictor, supervised by offline annotations from Vision-Language Models (VLMs), to route action chunks to experts specialized for specific behavioral phases. To ensure robust assignment, we propose a dual contrastive alignment strategy that grounds multi-modal observations in language-defined skill semantics (Inter-modal) while enforcing routing consistency across visually distinct but functionally related behaviors (Intra-modal). Our approach outperforms representative diffusion and MoE-based baselines on multi-task benchmarks with significantly improved parameter efficiency and demonstrates effective compositional transfer to novel tasks through parameter-efficient fine-tuning. Project website: https://deng-cy20.github.io/SMoDP/
Chengyu Deng, Guanqi Chen, Yizhou Chen +4
The University of Hong Kong · Southern University of Science and Technology