Advantage-Guided Diffusion for Model-Based Reinforcement Learning

Daniele Foffano; Arvid Eriksson; David Broman; Karl H. Johansson; Alexandre Proutiere

cs.AIApr 10, 2026

Advantage-Guided Diffusion for Model-Based Reinforcement Learning

Authors: Daniele Foffano, Arvid Eriksson, David Broman, Karl H. Johansson, Alexandre Proutiere

Organizations: Division of Decision and Control Systems of the EECS School at KTH Royal Institute of Technology, Stockholm, Sweden. · Digital Futures · Division of Software and Computer Systems of the EECS School at KTH Royal Institute of Technology, Stockholm, Sweden.

Abstract

Model-based reinforcement learning (MBRL) with autoregressive world models suffers from compounding errors, whereas diffusion world models mitigate this by generating trajectory segments jointly. However, existing diffusion guides are either policy-only, discarding value information, or reward-based, which becomes myopic when the diffusion horizon is short. We introduce Advantage-Guided Diffusion for MBRL (AGD-MBRL), which steers the reverse diffusion process using the agent's advantage estimates so that sampling concentrates on trajectories expected to yield higher long-term return beyond the generated window. We develop two guides: (i) Sigmoid Advantage Guidance (SAG) and (ii) Exponential Advantage Guidance (EAG). We prove that a diffusion model guided through SAG or EAG allows us to perform reweighted sampling of trajectories with weights increasing in state-action advantage-implying policy improvement under standard assumptions. Additionally, we show that the trajectories generated from AGD-MBRL follow an improved policy (that is, with higher value) compared to an unguided diffusion model. AGD integrates seamlessly with PolyGRAD-style architectures by guiding the state components while leaving action generation policy-conditioned, and requires no change to the diffusion training objective. On MuJoCo control tasks (HalfCheetah, Hopper, Walker2D and Reacher), AGD-MBRL improves sample efficiency and final return over PolyGRAD, an online Diffuser-style reward guide, and model-free baselines (PPO/TRPO), in some cases by a margin of 2x. These results show that advantage-aware guidance is a simple, effective remedy for short-horizon myopia in diffusion-model MBRL.

Figures & tables

Fig. 1 : MDP illustrating the negative effects of short-sighted planning.

Figure 2

Fig. 2 : Training curves for the Mujoco environments: HalfCheetah, Hopper, Walker and Reacher. Shaded areas indicate the standard error. The curves have been smoothed by a running average for better readability.

	Hopper	HalfCheetah	Walker	Reacher
AGD-MBRL (EAG)	3328.61 $\pm$ 39.45	4864.80 $\boldsymbol{\pm}$ 156.83	3794.24 $\boldsymbol{\pm}$ 165.03	-3.90 $\boldsymbol{\pm}$ 0.13
AGD-MBRL (SAG)	3268.42 $\pm$ 24.68	4642.25 $\boldsymbol{\pm}$ 162.52	3844.67 $\boldsymbol{\pm}$ 361.58	-3.87 $\boldsymbol{\pm}$ 0.37
PolyGRAD + Diffuser Guide	3353.97 $\boldsymbol{\pm}$ 25.30	4188.43 $\pm$ 489.66	3777.73 $\pm$ 294.98	-3.96 $\pm$ 0.40
PolyGRAD	3346.99 $\pm$ 52.39	$3879.16\pm 626.40$	$3489.48\pm 456.70$	$-4.48\pm 0.13$
PPO	$2998.90\pm 432.28$	$2408.20\pm 546.33$	$1894.03\pm 349.06$	$-5.17\pm 0.57$
TRPO	$3270.27\pm 273.04$	$2014.91\pm 539.64$	$3090.80\pm 267.79$	$-6.22\pm 0.85$

TABLE I : Final training return for MuJoCo continuous-control tasks.

Model-Based RL

Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.

See all 245 papers

Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
Dec 1, 2025Sebastian Sanokowski, Kaustubh Patil, Majid KhadivRL Control Diffusion Policy
Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University Munich · Practical Project Student Exchange Program, Technical University Munich · MIT World Peace University
Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance
May 28, 2026Shutong Ding, Zejia Zhong, Zhongyi Wang +4Diffusion Guidance Robotic RL
1ShanghaiTech University.
Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View
Aug 14, 2026Yixian Xu, Yuanrui Zhang, Shengjie Luo +2Diffusion Model Alignment Policy Gradient Methods
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · ByteDance Seed

Dec 1, 2025J/K move · Enter open · S save

Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning

Sebastian Sanokowski, Kaustubh Patil, Majid Khadiv

Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University Munich · Practical Project Student Exchange Program, Technical University Munich · MIT World Peace University

RL Control Diffusion Policy Maximum Entropy RL RL for Diffusion Models Policy Learning

Open paper AskPDF arXiv

Diffusion models provide an expressive framework for sampling from complex, unnormalized distributions. In this work, we extend Maximum Entropy Reinforcement Learning (ME-RL) to diffusion-based policies by introducing Diffusion-Augmented Markov Decision Processes (DA-MDPs). DA-MDPs interpret each reverse-diffusion transition as an individual reinforcement-learning decision, while only the final denoised action is executed in the environment. Our DA-MDPs follow from a principled derivation based on the variational-inference formulation of ME-RL. By augmenting policy and target trajectories with intermediate diffusion variables, we obtain a tractable reverse-KL upper bound via the data-processing inequality. This bound decomposes across denoising transitions, yielding diffusion-augmented variants of soft rewards, value functions, and local policy objectives. This provides a general framework for adapting ME-RL algorithms to diffusion policies while differentiating through only one diffusion transition at a time. We instantiate the framework with PPO, REPPO, and a maximum-entropy extension of WPO. Experiments demonstrate improved continuous-control performance, benefits from additional diffusion steps, and memory-efficient training. On the StackCube and PushT manipulation tasks, DA-MDP methods learn alternative successful strategies from the same initial state and achieve higher success rates and generally higher success-weighted mode entropy than the Gaussian ME-RL baseline. We also demonstrate successful training when using action chunking.

Abstract

Model-Based RL

Explore similar work

Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning

Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View