In offline reinforcement learning, diffusion models fill two roles with complementary strengths. Typically guided by a critic, a diffusion policy is value-aware and favors high-return, behavior-supported actions, but it is short-sighted: it never lays out the states it intends to visit. A diffusion planner is far-sighted, synthesizing such a long-horizon trajectory, but it is only behavior-driven at execution, realizing each planned transition using only offline behavior.Each is strong exactly where the other is weak. We introduce DPDP to bring them together at the planner's execution step, where a planned transition becomes an action. Reading the planner's inverse-dynamics executor as a posterior over actions: the environment's transition likelihood relates an action to the planned next state, while the behavioral prior determines its weight under the offline data. Replacing this prior with a value-optimized diffusion policy preserves the planned transition likelihood and favors valuable actions that can serve the plan. We target the resulting clean-action distribution in two ways. A training-free sampler composes policy and inverse-dynamics scores at matched action noise levels and applies Langevin correction; this route uses a shared action noise schedule and an approximately common terminal prior. A preference objective instead amortizes the transfer into a single diffusion executor. Across D4RL, DPDP improves the planner it builds on, staying competitive with reactive diffusion policies on locomotion and leading on long-horizon navigation.
Figures & tables
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
MuJoCo
AntMaze
Maze2D
Batch size
128
128
128
State–action generation
Separate
Separate
Separate
Planner guidance
MCSS
MCSS
MCSS
Advantage weighting
True
False
False
Discount γ
0.997
1.0
1.0
Planner backbone
Transformer
Transformer
Transformer
Appendix
Table A1: Per-suite configuration of the base models, from the released configuration: the Diffusion-Veteran planner, its diffusion inverse-dynamics executor, and the value critic used for candidate selection.
Setting
MuJoCo
AntMaze
Maze2D
Value strength β
1
1
1
DPDP-inf corrector budget M
1
1
1
DPDP-ft fine-tuning steps
2×105
2×105
2×105
DPDP-inf additional training
none
none
none
Appendix
Table A2: Transfer hyperparameters for DPDP-ft and DPDP-inf. Execution uses the plain conditional inverse-dynamics branch; we use full value transfer ( β=1 ) and a single corrector step ( M=1 ) throughout.
Task
DV +
N=4
N=16
DPDP-inf
DPDP-ft
maze2d-umaze-v1
129.2 ± 1.4
133.7 ± 4.8
136.8 ± 4.3
142.3 ± 1.5
145.7 ± 1.5
maze2d-medium-v1
146.3 ± 1.0
147.1 ± 4.1
152.0 ± 3.3
155.5 ± 1.0
152.7 ± 1.0
maze2d-large-v1
203.1 ± 1.4
195.9 ± 4.5
200.2 ± 6.9
209.0 ± 1.5
203.7 ± 1.4
halfcheetah-medium-v2
50.8 ± 0.1
50.8 ± 0.1
51.0 ± 0.1
50.9 ± 0.1
51.9 ± 0.1
halfcheetah-medium-replay-v2
46.0 ± 0.3
46.6 ± 0.1
46.7 ± 0.1
46.8 ± 0.1
47.3 ± 0.2
halfcheetah-medium-expert-v2
92.7 ± 0.2
92.9 ± 0.1
93.7 ± 0.5
94.6 ± 0.3
92.0 ± 0.5
Appendix
Table A4: Q-reranking baseline beside the base planner and both transfer variants. DV + is the schedule-matched base planner, and DPDP-ft and DPDP-inf the two transfer realizations (all from Table A3 ); N=4,16 give the number of inverse-dynamics candidates the critic ranks (mean ± standard error). The un-reranked control N=1 coincides with DV + and is omitted. Best mean per row in bold .
β
DPDP-inf
DPDP-ft
halfcheetah-medium-v2
∞
50.8
50.9
2
50.9
50.7
1
50.9
51.9
halfcheetah-medium-replay-v2
∞
46.0
46.2
Appendix
Table A5: Value-strength β sweep on the HalfCheetah tasks. β=∞ zeros the value branch and recovers the base planner, while β=1 is full value transfer. The default operating point used in the main results is β=1 .
β
DPDP-inf
DPDP-ft
hopper-medium-v2
∞
82.3
83.2
2
83.9
83.3
1
84.3
83.2
hopper-medium-replay-v2
∞
91.2
91.1
Appendix
Table A6: Value-strength β sweep on the Hopper tasks. Conventions as in Table A5 .
β
DPDP-inf
DPDP-ft
walker2d-medium-v2
∞
83.3
83.4
2
83.3
84.8
1
85.5
85.0
walker2d-medium-replay-v2
∞
85.7
85.6
Appendix
Table A7: Value-strength β sweep on the Walker2d tasks. Conventions as in Table A5 .
Task
M=0
M=1
M=3
M=5
halfcheetah-medium-v2
50.8
50.9
50.8
50.8
halfcheetah-medium-replay-v2
46.4
46.8
46.8
46.4
halfcheetah-medium-expert-v2
94.0
94.6
94.8
92.9
hopper-medium-v2
83.7
84.3
84.5
84.4
hopper-medium-replay-v2
92.2
92.2
91.2
91.5
hopper-medium-expert-v2
110.3
110.7
110.9
109.9
Appendix
Table A8: DPDP-inf as a function of the Langevin corrector budget M (normalized score; M=0 is the uncorrected composition). Suite-average rows are labeled in bold ; the best budget in each row is bold .
Task
M=0
M=1
M=3
M=5
marginal
DPDP-ft
halfcheetah-medium-v2
7.7
8.6
9.0
10.3
6.0
4.5
halfcheetah-medium-replay-v2
7.7
7.2
8.6
13.6
5.8
2.4
halfcheetah-medium-expert-v2
6.8
9.4
10.0
10.4
6.0
3.3
hopper-medium-v2
7.9
7.2
8.7
12.7
4.0
3.0
hopper-medium-replay-v2
6.4
7.0
10.7
11.7
4.4
2.0
hopper-medium-expert-v2
7.9
7.4
8.9
10.3
5.9
3.3
Appendix
Table A9: Wall-clock per evaluation run (minutes) on locomotion. M=0 – 5 are DPDP-inf with that many Langevin corrections; marginal is the plain single-model sampler; DPDP-ft is the amortized student. Lowest per row in bold .
Diffusion models have shown promising capabilities in trajectory generation for planning in offline reinforcement learning (RL). However, conventional diffusion-based planning methods often fail to account for the fact that generating trajectories in RL requires unique consistency between transitions to ensure coherence in real environments. This oversight can result in considerable discrepancies between the generated trajectories and the underlying mechanisms of a real environment. To address this problem, we propose a novel diffusion-based planning method, termed as Diffusion Modulation via Environment Mechanism Modeling (DMEMM). DMEMM modulates diffusion model training by incorporating key RL environment mechanisms, particularly transition dynamics and reward functions. Experimental results demonstrate that DMEMM achieves state-of-the-art performance for planning with offline reinforcement learning.
Hanping Zhang, Yuhong Guo
School of Computer Science, Carleton University, Ottawa, Canada
Offline reinforcement learning enables policy learning from fixed datasets without additional environment interaction, making it appealing for safety-critical applications where online exploration is costly or unsafe. Diffusion-based decision-making methods have recently achieved strong performance in offline RL by modeling rich, multimodal trajectory distributions. However, existing diffusion planners are typically risk-neutral and therefore may overlook rare but catastrophic outcomes that are crucial in real-world deployment. In this work, we propose RS-Diffuser, a risk-sensitive offline diffusion planning framework that combines diffusion-based trajectory generation with distributional value critics. RS-Diffuser learns a diffusion planner over future state trajectories, a separate inverse dynamics model for action decoding, and a Monte Carlo distributional critic that estimates the full return distribution of candidate plans through quantile regression. At sampling time, we incorporate a risk-sensitive guidance signal into the denoising process, using gradients computed from tail-aware objectives such as Conditional Value at Risk to steer generation toward desired risk profiles. As a result, a single trained model can flexibly produce risk-averse, risk-neutral, or risk-seeking behaviors by changing only the inference-time risk parameter. Extensive experiments on risk-sensitive D4RL and risky robot navigation benchmarks demonstrate that RS-Diffuser achieves state-of-the-art performance, improving both overall return and worst-case robustness while reducing safety violations.
Diffusion-based trajectory planners have shown strong performance in offline reinforcement learning, but their iterative denoising process often incurs high inference cost. Consistency-based planners reduce the number of sampling steps, yet they typically rely on a two-stage teacher--student distillation pipeline that increases training cost and may introduce instability. We propose Shortcut Trajectory Planning (STP), an offline model-based reinforcement learning framework that incorporates shortcut models as efficient trajectory generators. STP trains a conditional shortcut trajectory model in a single stage, supports adjustable one-step and few-step inference through step-size conditioning, and selects candidate plans using a critic augmented with feasibility-aware correction. Across standard D4RL benchmarks, including locomotion, navigation, manipulation, and dexterous control tasks, STP achieves strong performance while simplifying the training pipeline for fast generative planning.
Guanquan Wang, Yoshimasa Tsuruoka
Department of Information and Communication Engineering · The University of Tokyo