Diffusion Policy through Conditional Proximal Policy Optimization

Ben Liu; Shunpeng Yang; Hua Chen

cs.LGMar 5, 2026

Diffusion Policy through Conditional Proximal Policy Optimization

Authors: Ben Liu, Shunpeng Yang, Hua Chen

Organizations: Southern University of Science and Technology, Shenzhen, China · Limx Dynamics · Hong Kong University of Science and Technology, Hongkong, China · Zhejiang University-University of Illinois Urbana-Champaign Institute, Haining, China

Abstract

Reinforcement learning (RL) has been extensively employed in a wide range of decision-making problems, such as games and robotics. Recently, diffusion policies have shown strong potential in modeling multi-modal behaviors, enabling more diverse and flexible action generation compared to the conventional Gaussian policy. Despite various attempts to combine RL with diffusion, a key challenge is the difficulty of computing action log-likelihood under the diffusion model. This greatly hinders the direct application of diffusion policies in on-policy reinforcement learning. Most existing methods calculate or approximate the log-likelihood through the entire denoising process in the diffusion model, which can be memory- and computationally inefficient. To overcome this challenge, we propose a novel and efficient method to train a diffusion policy in an on-policy setting that requires only evaluating a simple Gaussian probability. This is achieved by aligning the policy iteration with the diffusion process, which is a distinct paradigm compared to previous work. Moreover, our formulation can naturally handle entropy regularization, which is often difficult to incorporate into diffusion policies. Experiments demonstrate that the proposed method produces multimodal policy behaviors and achieves superior performance on a variety of benchmark tasks in both IsaacLab and MuJoCo Playground.

Figures & tables

Figure 1: Overview of the proposed method. We align reinforcement learning policy iteration with the diffusion generative process through a novel policy parameterization. Unlike standard diffusion, where the probability density function is updated via a pre-defined SDE Euler–Maruyama step, our method employs conditional PPO to determine the Gaussian kernel for policy updates.

\max_{\theta}\mathbb{E}_{\{s,a_{0},a\}}\left[\text{CLIP}\left(r_{\theta}\hat{A}(s,a)\right)+\alpha\mathcal{H}(p_{\theta})-\beta\mathcal{R}_{\theta}\right]

Algorithm 1 DP-CPPO

Figure 2: Left. Multi-Goal environments, where different trajectories starting from the origin under the diffusion policy are illustrated. Contour lines in the figure are drawn based on the distance cost. Right. The positions after taking the first action from different saddle points are shown, visualizing the distribution of the policy $\pi(a|s)$ . The diffusion policy exhibits multimodal behavior, whereas the Gaussian policy collapses to near-zero movement due to the averaging effect of opposite goals.

Method	Initial Position
Method	(0.0,0.0)	(2.5,2.5)	(-2.5,2.5)
Gaussian PPO	-41.80	-16.47	-6.47
DP-CPPO	-11.94	1.82	1.49

Table 1: Reward at saddle points, higher is better.

Method	PPO	DP-CPPO (8)	DP-CPPO (16)
Time (min)	4.68	8.05 (+71.9%)	9.31 (+99.9%)
Mem (MB)	4202	4306 (+2.5%)	4306 (+2.5%)

Table 2: Computational cost for Ant task in IsaacLab. Training time and GPU Memory consumption are displayed, less is better. DP-CPPO (

k

) indicates DP-CPPO with

k

flow steps.

Figure 3: Rewards on Playground FingerSpin.

Figure 4: Training rewards across eight environments in IsaacLab. Results show the mean and standard deviation over 5 runs with different seeds (higher is better). For better visualization, we have smoothed the reward data. The proposed method outperforms Gaussian PPO in most tasks.

	Ant	Franka	Quadcopter	AnymalD	H1	Go2	Navigation	G1
Flow	220.74 $\pm$ 33.31	159.43 $\pm$ 0.37	141.34 $\pm$ 0.77	25.33 $\pm$ 0.10	29.40 $\pm$ 0.41	34.71 $\pm$ 0.15	4.58 $\pm$ 0.10	43.89 $\pm$ 1.32
Flow+Res	248.47 $\pm$ 37.03	159.48 $\pm$ 0.42	141.88 $\pm$ 0.36	25.34 $\pm$ 0.14	29.41 $\pm$ 0.42	34.97 $\pm$ 0.28	4.59 $\pm$ 0.09	44.17 $\pm$ 1.21
PPO	199.09 $\pm$ 10.58	156.46 $\pm$ 2.28	138.44 $\pm$ 4.71	24.64 $\pm$ 0.07	30.63 $\pm$ 0.35	32.56 $\pm$ 0.20	4.31 $\pm$ 0.11	28.48 $\pm$ 1.28
DPPO	192.73 $\pm$ 16.16	7.76 $\pm$ 6.34	141.64 $\pm$ 0.22	23.65 $\pm$ 0.46	9.07 $\pm$ 3.70	16.59 $\pm$ 0.00	2.13 $\pm$ 2.49	2.87 $\pm$ 1.43

Table 3: Evaluating rewards on IsaacLab tasks, higher is better.

	BallCup	Cheetah	FingerSpin	PointMass	Reach-E	Reach-H	WlkWalk	WlkRun
Flow	846.8 $\pm$ 57.7	822.6 $\pm$ 26.0	868.2 $\pm$ 92.0	860.3 $\pm$ 6.0	918.9 $\pm$ 11.4	857.2 $\pm$ 24.7	865.9 $\pm$ 89.0	641.4 $\pm$ 27.3
Flow+Res	866.2 $\pm$ 44.6	848.1 $\pm$ 19.1	912.5 $\pm$ 49.1	853.1 $\pm$ 2.9	898.4 $\pm$ 34.5	873.4 $\pm$ 28.7	894.0 $\pm$ 43.5	660.5 $\pm$ 22.3
FPO	879.4 $\pm$ 45.7	890.2 $\pm$ 7.8	655.2 $\pm$ 147.9	712.6 $\pm$ 85.3	902.5 $\pm$ 10.4	870.1 $\pm$ 35.8	880.1 $\pm$ 28.4	613.2 $\pm$ 29.2
DPPO	558.3 $\pm$ 319.7	401.6 $\pm$ 17.1	465.5 $\pm$ 57.1	546.3 $\pm$ 136.4	828.2 $\pm$ 29.3	742.1 $\pm$ 49.8	116.2 $\pm$ 6.4	49.7 $\pm$ 4.6

Table 4: Evaluating rewards on Playground tasks, higher is better.

Panel of Figure 5 — Figure 5: Ablation study on score-based regularization.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Training rewards across eight environments in Playground. Results show the mean and standard deviation over 5 runs with different seeds (higher is better). For better visualization, we have smoothed the reward data. Here, ‘epoch/batch’ denotes how many times each rollout is reused; smaller values indicate higher data efficiency. Our method achieves superior data efficiency.

Panel of Figure 7 — Figure 7: Ablation study on the number of flow-matching training epochs. (a) Reward difference between the optimal policy and the flow-matched policy (denoted by $(\pi^{*}-\pi_{\text{fitting}})$ ), showing that the fitting error gradually decreases and converges to a small value. (b) Training rewards of the flow-matched policy across different epoch settings, all of which converge to nearly the same final performance.

Panel of Figure 8 — Figure 8: (a) Reward difference between the sampling policy and the reference policy. The consistently positive values indicate that the sampling policy is better than the reference policy throughout training. (b) Training rewards with and without the EMA trick. Although the final performance is similar, the version without EMA succeeds in only 1 out of 6 runs, indicating it has reduced stability.

Panel of Figure 9 — Figure 9: Joint ablation on entropy and score-based regularization. In the figure, the check mark indicates that a suitable parameter is selected, while the cross indicates that this option is not applied.

Panel of Figure 10 — Figure 10: Entropy in Multi-Goal environment. The trends of the two entropy measures are generally consistent.

Figure 11: Reward on Isaac-Lift-Cube-Franka-v0 with different numbers of flow steps. All configurations achieve similar final performance.

Hyperparameters	Value
Flow steps	16
Learning rate	0.001
Ema rate	0.999
Mini batches	4
Training epochs	15
Optimizer	Adam

Appendix

Table 5: Hyperparameters for Flow Matching (IsaacLab and Playground).

Hyperparameters	DP-CPPO	PPO
GAE Smoothing	0.95	0.95
Desired kl	0.01	0.01
Ratio clip	0.2	0.2
Max gradient clip	1.0	1.0
Value loss scale	1.0	1.0
Value clip	True	True

Appendix

Table 6: Hyperparameters for Isaac-Ant-v0.

Hyperparameters	DP-CPPO	PPO
GAE Smoothing	0.95	0.95
Desired kl	0.008	0.01
Ratio clip	0.2	0.2
Max gradient clip	1.0	1.0
Value loss scale	1.0	1.0
Value clip	True	True

Appendix

Table 7: Hyperparameters for Isaac-Lift-Cube-Franka-v0.

Hyperparameters	DP-CPPO	PPO
GAE Smoothing	0.95	0.95
Desired kl	0.01	0.01
Ratio clip	0.2	0.2
Max gradient clip	1.0	1.0
Value loss scale	1.0	1.0
Value clip	True	True

Appendix

Table 8: Hyperparameters for Isaac-Quadcopter-Direct-v0.

Hyperparameters	DP-CPPO	PPO
GAE Smoothing	0.95	0.95
Desired kl	0.01	0.01
Ratio clip	0.2	0.2
Max gradient clip	1.0	1.0
Value loss scale	1.0	1.0
Value clip	True	True

Appendix

Table 9: Hyperparameters for Isaac-Velocity-Flat-Anymal-D-v0.

Hyperparameters	DP-CPPO	PPO
GAE Smoothing	0.95	0.95
Desired kl	0.01	0.01
Ratio clip	0.2	0.2
Max gradient clip	1.0	1.0
Value loss scale	1.0	1.0
Value clip	True	True

Appendix

Table 10: Hyperparameters for Isaac-Velocity-Rough-H1-v0.

Hyperparameters	DP-CPPO	PPO
GAE Smoothing	0.95	0.95
Desired kl	0.01	0.01
Ratio clip	0.2	0.2
Max gradient clip	1.0	1.0
Value loss scale	1.0	1.0
Value clip	True	True

Appendix

Table 11: Hyperparameters for Isaac-Velocity-Rough-Unitree-Go2-v0.

Hyperparameters	DP-CPPO	PPO
GAE Smoothing	0.95	0.95
Desired kl	0.01	0.01
Ratio clip	0.2	0.2
Max gradient clip	1.0	1.0
Value loss scale	1.0	1.0
Value clip	True	True

Appendix

Table 12: Hyperparameters for Isaac-Navigation-Flat-Anymal-C-v0.

Hyperparameters	DP-CPPO	PPO
GAE Smoothing	0.95	0.95
Desired kl	0.01	0.01
Ratio clip	0.2	0.2
Max gradient clip	1.0	1.0
Value loss scale	1.0	1.0
Value clip	True	True

Appendix

Table 13: Hyperparameters for Isaac-Velocity-Rough-G1-v0.

Hyperparameters	Value
GAE Smoothing	0.95
Desired kl	0.01
Ratio clip	0.2
	0.05 (BallinCup)
Max gradient clip	1.0
Value loss scale	1.0

Appendix

Table 14: Hyperparameters for Playground tasks.

Diffusion Policy

Diffusion policies generate multimodal action sequences by iteratively denoising noise, enabling imitation, planning, and control in robotics.

See all 205 papers

Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching
Jun 21, 2026Serge Thilges, Onur Celik, Denis Blessing +2Diffusion Policy Maximum Entropy RL
Autonomous Learning Robots, Karlsruhe Institute of Technology
Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning
Sep 7, 2026Mahmoud Selim, Cristina Cipriani, Karl H. JohanssonDiffusion Policy RL for Diffusion Models
TRATON CV AB, Stockholm, Sweden · KTH Royal Institute of Technology, Stockholm, Sweden
DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models
May 14, 2026Quanhao Li, Junqiu Yu, Kaixun Jiang +7Diffusion Model Distillation Multi-Task RL
Fudan University, China · Alibaba Group, China

Jun 21, 2026J/K move · Enter open · S save

Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching

Serge Thilges, Onur Celik, Denis Blessing, Emiliyan Gospodinov+1

Autonomous Learning Robots, Karlsruhe Institute of Technology

Diffusion Policy Maximum Entropy RL RL for Diffusion Models

Open paper AskPDF arXiv

Diffusion policies have recently emerged as a powerful paradigm for representing complex action distributions in reinforcement learning (RL). However, their application to online RL remains limited by the challenge of scalable training in the absence of ground-truth data, where standard optimization techniques such as score matching are not directly applicable. In this work, we introduce a highly efficient algorithm for optimizing diffusion policies by leveraging recent advances in stochastic optimal control. Our approach is based on adjoint matching, which enables simulation-free training and circumvents the need for explicit likelihood estimation or costly backpropagation through the diffusion process. Furthermore, we propose several extensions that improve the robustness and stability of the method in practical settings. Empirical results demonstrate that our approach achieves competitive performance while significantly reducing computational overhead, making diffusion policies more viable for online RL scenarios.

Abstract

Diffusion Policy

Explore similar work

Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching

Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning

DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models