RL for Diffusion Models

The field studies how to adapt diffusion generators using task rewards, human preferences, or environment feedback rather than relying only on imitation or supervised denoising. Common approaches treat denoising steps as policy actions and use PPO or GRPO policy gradients, reward-weighted flow or score matching, distribution matching, and critic-guided control; distillation can make these updates cheaper. Applications span image and video alignment, reasoning in discrete diffusion language models, and robot policies, evaluated on benchmarks such as GenEval, GSM8K, and MuJoCo tasks. Key tensions include estimating useful learning signals across denoising steps, preventing reward hacking without losing diversity or fidelity, and achieving gains without expensive rollouts.