Diffusion models have shown remarkable progress in AI-generated content, including images and videos. In this paper, we explore whether diffusion models can generate the parameters of policy networks in reinforcement learning (RL), and whether the generated policies exhibit generalization capabilities. To achieve this, we propose Diffusion-based Conditional Policy Network Parameter Generation (DPPG), which consists of a Transformer-based autoencoder and a task-conditioned diffusion model. The autoencoder achieves a high compression rate by patchifying network parameters, while the task-conditioned diffusion model generates corresponding parameters conditioned on different environments. Our experiments demonstrate that, with a single unified DPPG model, we can synthesize high-quality policies across a diverse set of tasks from different environment domains, including Classic Control, Atari games, and MuJoCo. Additionally, our method can generate networks with a larger number of parameters compared to previous approaches. Furthermore, we find that in previously unseen environments, DPPG can generate parameters that achieve faster convergence and reach competitive performance, showcasing the potential future applications of parameter generation via diffusion.
Figures & tables
Figure 1: An overview of the proposed framework. (a) Prepare the parameters of policy networks trained in various reinforcement learning environments. (b) Train a Transformer-based autoencoder to compress the parameters. (c) Train a diffusion model to generate the compressed latent representations. (d) Reconstruct the final policy network parameters from random noise during inference.
Environment
Parameter number
Original
pdiff
DPPG (Ours)
Atari
Alien
9234
674.78±48.17
856.55±238.60
735.0±50.69
Amidar
5130
761.22±12.34
583.51±220.35
772.68±14.94
Assault
3591
5562.01±739.71
153.93±401.77
6500.79±638.09
Asteroids
7182
1160.70±99.18
865.62±307.34
1185.55±97.08
BattleZone
9234
23556.0±1743.23
23635.0±4832.83
28113.0±2029.09
BeamRider
4617
4183.13±455.50
4177.47±828.9
4364.60±517.49
Table 1: Performance in Single Environments. We train DPPG within a single environment and test within the same setting. "Parameter Number" refers to the size of the network parameters. "Original" denotes the returns of the baseline RL model. For each environment, we generate 200 sets of network parameters and report the mean and variance of their returns.
Environment
Parameter number
Original
DPPG (Ours)
Atari
Alien
9234
674.78±48.17
727.14±41.18
Amidar
5130
761.22±12.34
761.79±13.27
Assault
3591
5562.01±739.71
5140.74±654.34
Asteroids
7182
1160.70±99.18
1208.31±105.47
BattleZone
9234
23556.0±1743.23
21253.5±2143.00
BeamRider
4617
4183.13±455.50
4511.57±414.10
Table 2: Performance Across the Entire Environments. We trained a unified DPPG model across all environments and conducted tests within their respective settings. "Parameter Number" refers to the size of the network parameters. "Original" denotes the returns of the baseline RL model. For each environment, we generate 200 sets of network parameters and report the mean and variance of their returns.
Environment
Parameter number
Original
DPPG (Ours)
Atari
Alien
1693362
763.28±82.81
799.75±98.65
Amidar
1684641
724.78±19.60
763.36±20.66
Assault
1683102
5270.33±696.26
5688.04±514.72
Asteroids
1686693
1198.2±112.04
1189.4±83.49
BattleZone
1693362
19131.5±1968.21
20725.0±2310.16
BeamRider
1688745
4437.67±606.95
4548.55±477.68
Table 3: Performance on Large Parameters. "Parameter Number" refers to the size of the network parameters. "Original" denotes the returns of the baseline RL model. For each environment, we generate 200 sets of network parameters and report the mean and variance of their returns.
Figure 2: Under the same GPU memory, DPPG can generate a network with approximately 100 times the number of parameters as pdiff.
Environment
Initialization
Fine-tuning
Random Init
Orthogonal Init
DPPG
Random Init
Orthogonal Init
DPPG
AirRaid
0
297.5
26.5
0
6080
11150
Defender
550
1165
1953
1180
30080
31190
DoubleDunk
−4.0
−4.2
−3.15
-2.8
-2.4
−0.4
Gopher
48
0
59
1300
924
1520
Phoenix
58
612
74.4
3934
4888
6524
Table 4: Performance on Unseen Environments. We report two sets of results: (a) performance without fine-tuning; (b) performance after fine-tuning.
Figure 3: Episode reward curve over fine-tuning steps in unseen environments, using generated parameters as initialization.
Environment
128
256
512
1024
2048
MountainCar
0.98
1.03
1.01
1.02
1.02
Ant
0.75
0.85
1.10
1.5
1.5
Assault
1.09
1.04
1.13
1.15
1.17
Table 5: Average scores with different autoencoder patch sizes
Figure 4: Impact of different K . K denotes the size of the training data. Increasing K leads to more stable generation outcomes.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Impact of different autoencoder patch sizes on generation performance in large parameter settings. Both the autoencoder and the diffusion model were trained for 20,000 and 80,000 iterations, respectively.
Environment
2
4
8
16
32
MountainCar
1.01
1.03
0.99
1.02
1.04
Ant
1.33
1.27
1.5
1.5
1.27
Assault
1.07
0.99
1.06
1.17
1.08
Appendix
Table 6: Average scores with different diffusion model patch sizes
Training Setting
Small Parameters
Large Parameters
K
200
200
batch size
200
200
Autoencoder
optimizer
AdamW
AdamW
learning rate
3e-4
3e-4
training iterations
20,000
20,000
Appendix
Table 7: Training settings comparison for small and large parameters.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
We propose Parameterized Diffusion Policy (PDP), a framework for learning diffusion policies conditioned on low-dimensional, continuous parameters embedded in a learned behavior manifold. By constructing this manifold so that distances between latent representations reflect the semantic similarity between physical trajectories, we transform diffusion from a mechanism for stochastic diversity into a precise and optimizable tool for behavior steering. Our approach enables smooth interpolation between known strategies and efficient adaptation to novel constraints without updating policy weights. We demonstrate that PDP significantly improves adaptation performance on complex multimodal benchmarks in both simulated and real-robot experiments compared to standard diffusion policies, particularly in scenarios requiring the synthesis of novel behaviors.
Renhao Zhang, Haotian Fu, Mingxi Jia +3
University of Massachusetts · 2Brown University · 3Harvard
Reinforcement learning (RL) has been extensively employed in a wide range of decision-making problems, such as games and robotics. Recently, diffusion policies have shown strong potential in modeling multi-modal behaviors, enabling more diverse and flexible action generation compared to the conventional Gaussian policy. Despite various attempts to combine RL with diffusion, a key challenge is the difficulty of computing action log-likelihood under the diffusion model. This greatly hinders the direct application of diffusion policies in on-policy reinforcement learning. Most existing methods calculate or approximate the log-likelihood through the entire denoising process in the diffusion model, which can be memory- and computationally inefficient. To overcome this challenge, we propose a novel and efficient method to train a diffusion policy in an on-policy setting that requires only evaluating a simple Gaussian probability. This is achieved by aligning the policy iteration with the diffusion process, which is a distinct paradigm compared to previous work. Moreover, our formulation can naturally handle entropy regularization, which is often difficult to incorporate into diffusion policies. Experiments demonstrate that the proposed method produces multimodal policy behaviors and achieves superior performance on a variety of benchmark tasks in both IsaacLab and MuJoCo Playground.
Ben Liu, Shunpeng Yang, Hua Chen
Southern University of Science and Technology, Shenzhen, China · Limx Dynamics · Hong Kong University of Science and Technology, Hongkong, China +1
Diffusion and flow models provide expressive policy classes for robotic and sequential decision-making systems. However, their action likelihoods are generally intractable, limiting the direct use of widely adopted policy optimization methods such as PPO and GRPO. We propose Doob Policy Optimization (DoobPO), a framework that leverages Doob's h-transform for likelihood-free reinforcement learning fine-tuning of generative policies. DoobPO separates policy improvement from generative policy learning. It first learns a lightweight ratio model that represents the desired policy update, and then distills the resulting ratio-tilted target policy into the generative policy via conditional drift matching. This formulation applies uniformly to both diffusion and flow policies and supports a broad family of policy improvement rules, including PPO and GRPO. We prove that the practical conditional drift matching objective is gradient-equivalent to the ideal Doob drift matching objective. We further show that the induced diffusion process solves an endpoint-constrained stochastic optimal control problem. This establishes that it achieves the minimum-energy modification required to realize the desired policy tilt. Across MuJoCo continuous-control benchmarks, DoobPO achieves strong performance against competitive diffusion- and flow-policy baselines, showing a principled and practical route to RL fine-tuning for generative policies.
Maojiang Su, Qijie Zhu, Yu Xiao +4
Department of Computer Science, Northwestern University · Center for Foundation Models and Generative AI, Northwestern University · Department of Statistics and Data Science, Northwestern University +1