Diffusion Guidance Is a Controllable Policy Improvement Operator
Organizations: UC Berkeley
Abstract
At the core of reinforcement learning is the idea of learning beyond the performance in the data. However, scaling such systems has proven notoriously tricky. In contrast, techniques from generative modeling have proven remarkably scalable and are simple to train. In this work, we combine these strengths, by deriving a direct relation between policy improvement and guidance of diffusion models. The resulting framework, CFGRL, is trained with the simplicity of supervised learning, yet can further improve on the policies in the data. On offline RL tasks, we observe a reliable trend -- increased guidance weighting leads to increased performance. Of particular importance, CFGRL can operate without explicitly learning a value function, allowing us to generalize simple supervised methods (e.g., goal-conditioned behavioral cloning) to further prioritize optimality, gaining performance for "free" across the board.
Figures & tables
| Task | AWR | Diffusion AWR | FQL | DDPG | CFGRL |
| pointmaze-large-navigate | |||||
| pointmaze-teleport-navigate | |||||
| antmaze-large-navigate | |||||
| antmaze-teleport-navigate | |||||
| humanoidmaze-large-navigate | |||||
| antsoccer-arena-navigate |
| Flat Policies | Hierarchical Policies | |||||||
| Task | BC | Flow BC | GCBC | Flow GCBC | CFGRL | HGCBC | Flow HGCBC | HCFGRL |
| pointmaze-medium-navigate | ||||||||
| pointmaze-large-navigate | ||||||||
| pointmaze-giant-navigate | ||||||||
| pointmaze-teleport-navigate | ||||||||
| antmaze-medium-navigate | ||||||||
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Learning rate | |
| Optimizer | Adam [ 38 ] |
| Gradient steps | |
| Minibatch size | |
| MLP dimensions | |
| Nonlinearity | GELU [ 33 ] |
| Hyperparameter | Value |
|---|---|
| Learning rate | |
| Optimizer | Adam [ 38 ] |
| Gradient steps | |
| Minibatch size | (states), (pixels) |
| MLP dimensions | |
| Nonlinearity | GELU [ 33 ] |
Policy Learning
Policy learning develops decision rules that choose actions or interventions to maximize outcomes from data, feedback, or interaction.
See all 501 papers