Symmetry-Aware Steering of Equivariant Diffusion Policies: Benefits and Limits

Minwoo Park; Junwoo Chang; Jongeun Choi; Roberto Horowitz

cs.LGDec 12, 2025

Symmetry-Aware Steering of Equivariant Diffusion Policies: Benefits and Limits

Authors: Minwoo Park, Junwoo Chang, Jongeun Choi, Roberto Horowitz

Organizations: Yonsei University, Seoul 03722, Republic of Korea · University of California, Berkeley, CA 94720, USA

Abstract

Equivariant diffusion policies (EDPs) combine the expressivity of diffusion models with the generalization and sample efficiency afforded by geometric symmetries. While steering these policies with reinforcement learning (RL) offers a promising fine-tuning mechanism beyond demonstration data, directly applying standard (non-equivariant) RL ignores the symmetries that EDPs are designed to exploit, leading to sample inefficiency and instability. We theoretically establish that the diffusion process of an EDP is equivariant, inducing a group-invariant latent-noise MDP which is well-suited for equivariant diffusion steering. Building on this, we introduce a principled symmetry-aware steering framework and compare standard, equivariant, and approximately equivariant RL strategies across tasks with varying degrees of symmetry. While we identify the practical boundaries of strict equivariance under symmetry breaking, exploiting symmetry during the steering yields substantial benefits---enhancing sample efficiency, preventing value divergence, and achieving strong policy improvements even when EDPs are trained from extremely limited demonstrations.

Figures & tables

Figure 1: Equivariance in the latent-noise space. An EDP

\pi_{\mathrm{edp}}

is equivariant with respect to state

s

and initial latent noise

w

. Top : For a grasping task, optimal latent noise

w_{1}

generates an optimal action

a_{1}

, while

w_{2}

leads to suboptimal action

a_{2}

. Bottom : Under a

90^{\circ}

rotation

g

, an equivariant steering policy trained to select

w_{1}

for

s

is guaranteed to select

gw_{1}

for

gs

, generating the optimal action

ga_{1}

Panel of Figure 2 — Figure 2: Task visualizations. (a) Lift : Lifting a red cube. (b) Stack D1 : Stacking a red cube onto a green cube. (c) Square D2 : Inserting a square nut into a peg. Initial poses are randomized per episode.

Task	Before	DSRL	Equi-	Approx-Equi-
Task	Steering	DSRL	DSRL	DSRL
Lift	0.617	0.808	0.840	0.820
Stack D1	0.568	0.740	0.728	0.800
Square D2	0.275	0.552	0.644	0.604
Square D2 (Image)	0.283	0.380	0.450	0.470

Table 1: Maximum evaluation success rates.

Figure 3: Performance comparison. Results are averaged over 5M environment steps, using five random seeds (two for image-based). Shaded regions denote standard error.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	Name	Dim	Rep
Common	gripper position & rotation	9	$\rho_{1}\oplus\rho_{0}\oplus\rho_{1}^{3}$
Common	gripper joint position	2	$\rho_{0}^{2}$
Lift	cube position & rotation	9	$\rho_{1}\oplus\rho_{0}\oplus\rho_{1}^{3}$
Lift	gripper to cube position $(x,y)$	2	$\rho_{1}$
Stack D1	cube position & rotation	9	$\rho_{1}\oplus\rho_{0}\oplus\rho_{1}^{3}$
Stack D1	gripper to cube position $(x,y,z)$	3	$\rho_{1}\oplus\rho_{0}$

Appendix

Table 2: State and action representations.

Diffusion Policy

Diffusion policies generate multimodal action sequences by iteratively denoising noise, enabling imitation, planning, and control in robotics.

See all 207 papers

Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View
Aug 14, 2026Yixian Xu, Yuanrui Zhang, Shengjie Luo +2Diffusion Model Alignment Policy Gradient Methods
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · ByteDance Seed
ReDiF: Resource-Efficient Few-Step Diffusion Distillation via Reinforcement Learning
Dec 28, 2025Amirhossein Tighkhorshid, Zahra Dehghanian, Hamid R. RabieeDiffusion Distillation RL for Diffusion Models
Computer Engineering Sharif University of Technology
From Noise to Control: Parameterized Diffusion Policies
May 29, 2026Renhao Zhang, Haotian Fu, Mingxi Jia +3Robot Policy Learning Robot Policy Adaptation
University of Massachusetts · 2Brown University · 3Harvard

Aug 14, 2026J/K move · Enter open · S save

Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View

Yixian Xu, Yuanrui Zhang, Shengjie Luo, Liwei Wang+1

State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · ByteDance Seed

Diffusion Model Alignment Policy Gradient Methods RL for Diffusion Models Variance Reduction

Open paper AskPDF arXiv

Reinforcement learning (RL) post-training provides a direct way to align diffusion models with human preferences and task-specific rewards. However, current RL algorithms for diffusion models remain fragmented: reverse-trajectory methods rely on discretized likelihood ratios, whereas forward-matching methods train on reward-labeled noising versions of the rollout samples. This paper shows that these seemingly different losses arise from a single path-space principle. Starting from the regularized diffusion-RL objective, we use importance sampling between sampling SDEs to obtain an explicit policy-gradient estimator on trajectory space. The estimator contains the stochastic Itô integral underlying Flow-GRPO-type updates; we derive an equivalent variance-reduced value-gradient form that recovers the forward-matching structure of AWM and DiffusionNFT. This identifies the empirical gap between these method families as a variance-reduction effect rather than a difference in RL principle. The derivation yields a unified design space organized by value-gradient estimation, weight functions, and sampling choices. Within this space, we propose a multi-sample KDE value-gradient estimator that reuses rollout groups, together with scale-bounded weight families that retain stable existing recipes while excluding singular ones. Experiments on SD3.5-M and Qwen-Image models validate the variance-reduction explanation and show that the resulting recipe improves over prior diffusion-RL baselines.

Abstract

Diffusion Policy

Explore similar work

Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View

ReDiF: Resource-Efficient Few-Step Diffusion Distillation via Reinforcement Learning

From Noise to Control: Parameterized Diffusion Policies