Striking a balance between efficiency and transparent motion is a core challenge in human-robot collaboration, as highly expressive movements often incur unnecessary time and energy costs. In collaborative environments, legibility allows a human observer a better understanding of the robot's actions, increasing safety and trust. However, these behaviors result in sub-optimal and exaggerated trajectories that are redundant in low-ambiguity scenarios where the robot's goal is already obvious. To address this trade-off, we propose Style-Conditioned Diffusion Policy (SCDP), a modular framework that constrains the trajectory generation of a pre-trained diffusion model toward either legibility or efficiency based on the environment's configuration. Our method utilizes a post-training pipeline that freezes the base policy and trains a lightweight scene encoder and conditioning predictor to modulate the diffusion process. At inference time, an ambiguity detection module activates the appropriate conditioning, prioritizing expressive motion only for ambiguous goals and reverting to efficient paths otherwise. We evaluate SCDP on manipulation and navigation tasks, and results show that it enhances legibility in ambiguous settings while preserving optimal efficiency when legibility is unnecessary, all without retraining the base policy.
Figures & tables
Figure 1: Style-Conditioned Diffusion Policy is an offline imitation learning framework that allows for motion conditioning depending on the environment’s context. In ambiguous scenes (top), SCDP produces intent-expressive motion to resolve goal ambiguity. When ambiguity is low (bottom), it prioritizes task efficiency, avoiding sub-optimal and exaggerated trajectories.
Figure 2: (a) The predictor module is integrated via a post-training pipeline where the base Diffusion Policy weights remain frozen. By training the lightweight MLP on a subset of expressive demonstrations, the module learns to specifically compensate for the residuals between the style-specific trajectories and the general paths the base model was originally trained to reproduce.(b) The conditioning from the predictor is only applied to the bottleneck of the diffusion U-Net using FiLM to denoise X at each timestep t .
Figure 3: (a) The environment state is passed through the ambiguity detection module to determine if the scene is spatially ambiguous and decide which conditioning should be used. (b) Visualization of the ellipse of ambiguity used for the scene’s classification. The scene is labeled as spatially ambiguous when g− falls inside the elliptical boundary.
Figure 4: Visualization of SCDP and baselines’ inferences in ambiguous (top) and non-ambiguous (bottom) scene configurations for the navigation task. While Diffusion Policy captures the entire data distribution and Legibility Diffuser collapses on the most legible mode, SCDP constrains its generation depending on the scene configuration.
Method
Spatial Ambiguity
No Spatial Ambiguity
Block Reach Evaluation
Diffusion Policy
0.52±0.06
0.61±0.06
Legibility Diffuser
0.61±0.09
0.43±0.10
Ours (SCDP)
0.58±0.08
0.74±0.06
Dataset
0.47±0.17
0.62±0.18
Navigation Evaluation
Table 1: Performance Comparison using Adaptive Transparency Score for the Block Reach and Navigation tasks.
Method
Spatial Ambiguity
No Spatial Ambiguity
Detachment
Efficiency
Detachment
Efficiency
Block Reach Evaluation
Diffusion Policy
0.52±0.06
0.67±0.06
0.43±0.09
0.65±0.09
Legibility Diffuser
0.89\pm 0.12\color[rgb]{0,0.6,0}{\uparrow}
0.13\pm 0.10\color[rgb]{1,0,0}{\downarrow}
0.67\pm 0.16\color[rgb]{0,0.6,0}{\uparrow}
0.31\pm 0.20\color[rgb]{1,0,0}{\downarrow}
Ours (SCDP)
0.70\pm 0.10\color[rgb]{0,0.6,0}{\uparrow}
0.43\pm 0.06\color[rgb]{1,0,0}{\downarrow}
0.42±0.10
0.80\pm 0.12\color[rgb]{0,0.6,0}{\uparrow}
Dataset
0.46±0.22
0.64±0.25
0.50±0.22
0.64±0.24
Table 2: Separated Detachment and Trajectory Efficiency Scores Comparison for Block Reach and Navigation tasks.
Figure 5: Real-world deployment of SCDP on a Franka Emika Panda robot for the Block Reach task. The image sequence (left to right) illustrates the model successfully generating an exaggerated, intent-expressive trajectory toward the target blue object to resolve spatial ambiguity relative to the distractor pink object.
Diffusion-based visuomotor policies operating directly in raw action spaces conflate scene comprehension with trajectory generation within a single denoising process. The resulting velocity field must simultaneously encode scene information and generate precise trajectories, increasing learning complexity and limiting performance on tasks demanding precise temporal coordination across multiple arms. To simplify this joint learning problem, we introduce Latent Diffusion Policy (LDP), a two-stage framework performing flow matching in a deliberately shaped latent space. By absorbing scene understanding into an observation-conditioned CVAE encoder, LDP concentrates the conditional distribution of each observation. Consequently, the flow model avoids implicitly resolving scene-dependent structures; instead, it generates within a pre-concentrated distribution featuring a smoother velocity field, simplifying learning from limited demonstrations. Furthermore, to capture temporal dependencies among latent tokens, LDP trains with per-token diffusion forcing and employs staircase inference sampling to resolve the resulting distributional mismatch. We also propose reconstruction FID (rFID) as a lightweight proxy predicting downstream task success solely from latent space statistics. On coordination-intensive tasks from RoboTwin 2.0, LDP outperforms DP3 by a substantial margin and transfers effectively to real-world bimanual deployments.
Diffusion policies have shown to be very efficient at learning complex, multi-modal behaviors for robotic manipulation. However, errors in generated action sequences can compound over time which can potentially lead to failure. Some approaches mitigate this by augmenting datasets with expert demonstrations or learning predictive world models which might be computationally expensive. We introduce Performance Predictive Guidance (PPGuide), a lightweight, classifier-based framework that steers a pre-trained diffusion policy away from failure modes at inference time. PPGuide makes use of a novel self-supervised process: it uses attention-based multiple instance learning to automatically estimate which observation-action chunks from the policy's rollouts are relevant to success or failure. We then train a performance predictor on this self-labeled data. During inference, this predictor provides a real-time gradient to guide the policy toward more robust actions. We validated our proposed PPGuide across a diverse set of tasks from the Robomimic and MimicGen benchmarks, demonstrating consistent improvements in performance.
Zixing Wang, Devesh K. Jha, Ahmed H. Qureshi +1
Department of Computer Science at Purdue University, IN 47907, USA. · Contribution was conducted while the author was at Mitsubishi Electric Research Laboratories. · Mitsubishi Electric Research Laboratories, Cambridge, MA 02139 USA.
Diffusion models sample effectively from high-dimensional, multimodal distributions, but their outputs may violate deployment constraints. For task-space robot policies, generated grasps, waypoints, or trajectories can be distributionally valid yet infeasible, violating reachability, collision-avoidance, or closed-loop executability requirements. This embodiment gap limits zero-shot deployment across robots, even when the task-space behavior itself is transferable. We propose an inference-time optimization framework that couples the behavior generation to physical feasibility by formulating diffusion guidance as a constrained optimization problem. Our key insight is to replace the sampling perturbation in the backward process with an optimized correction, allowing hard constraints or soft penalties to be imposed during sampling without the need to retrain the diffusion model, while keeping samples close to the learned prior. We evaluate the method on dexterous grasp synthesis with reachability and collision-avoidance constraints, and dynamic manipulation with controller-level trackability constraints. Across settings and robot embodiments, optimization-guided denoising matches the feasibility of projection- and gradient-guidance baselines while better preserving grasp quality, and improving controller-level executability and task success, with task success improving by up to 20pp. on dexterous grasping and 23pp. on visuomotor manipulation over the best baseline.