In offline reinforcement learning, diffusion models fill two roles with complementary strengths. Typically guided by a critic, a diffusion policy is value-aware and favors high-return, behavior-supported actions, but it is short-sighted: it never lays out the states it intends to visit. A diffusion planner is far-sighted, synthesizing such a long-horizon trajectory, but it is only behavior-driven at execution, realizing each planned transition using only offline behavior.Each is strong exactly where the other is weak. We introduce DPDP to bring them together at the planner's execution step, where a planned transition becomes an action. Reading the planner's inverse-dynamics executor as a posterior over actions: the environment's transition likelihood relates an action to the planned next state, while the behavioral prior determines its weight under the offline data. Replacing this prior with a value-optimized diffusion policy preserves the planned transition likelihood and favors valuable actions that can serve the plan. We target the resulting clean-action distribution in two ways. A training-free sampler composes policy and inverse-dynamics scores at matched action noise levels and applies Langevin correction; this route uses a shared action noise schedule and an approximately common terminal prior. A preference objective instead amortizes the transfer into a single diffusion executor. Across D4RL, DPDP improves the planner it builds on, staying competitive with reactive diffusion policies on locomotion and leading on long-horizon navigation.
Figures & tables
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
MuJoCo
AntMaze
Maze2D
Batch size
128
128
128
State–action generation
Separate
Separate
Separate
Planner guidance
MCSS
MCSS
MCSS
Advantage weighting
True
False
False
Discount γ
0.997
1.0
1.0
Planner backbone
Transformer
Transformer
Transformer
Appendix
Table A1: Per-suite configuration of the base models, from the released configuration: the Diffusion-Veteran planner, its diffusion inverse-dynamics executor, and the value critic used for candidate selection.
Setting
MuJoCo
AntMaze
Maze2D
Value strength β
1
1
1
DPDP-inf corrector budget M
1
1
1
DPDP-ft fine-tuning steps
2×105
2×105
2×105
DPDP-inf additional training
none
none
none
Appendix
Table A2: Transfer hyperparameters for DPDP-ft and DPDP-inf. Execution uses the plain conditional inverse-dynamics branch; we use full value transfer ( β=1 ) and a single corrector step ( M=1 ) throughout.
Task
DV +
N=4
N=16
DPDP-inf
DPDP-ft
maze2d-umaze-v1
129.2 ± 1.4
133.7 ± 4.8
136.8 ± 4.3
142.3 ± 1.5
145.7 ± 1.5
maze2d-medium-v1
146.3 ± 1.0
147.1 ± 4.1
152.0 ± 3.3
155.5 ± 1.0
152.7 ± 1.0
maze2d-large-v1
203.1 ± 1.4
195.9 ± 4.5
200.2 ± 6.9
209.0 ± 1.5
203.7 ± 1.4
halfcheetah-medium-v2
50.8 ± 0.1
50.8 ± 0.1
51.0 ± 0.1
50.9 ± 0.1
51.9 ± 0.1
halfcheetah-medium-replay-v2
46.0 ± 0.3
46.6 ± 0.1
46.7 ± 0.1
46.8 ± 0.1
47.3 ± 0.2
halfcheetah-medium-expert-v2
92.7 ± 0.2
92.9 ± 0.1
93.7 ± 0.5
94.6 ± 0.3
92.0 ± 0.5
Appendix
Table A4: Q-reranking baseline beside the base planner and both transfer variants. DV + is the schedule-matched base planner, and DPDP-ft and DPDP-inf the two transfer realizations (all from Table A3 ); N=4,16 give the number of inverse-dynamics candidates the critic ranks (mean ± standard error). The un-reranked control N=1 coincides with DV + and is omitted. Best mean per row in bold .
β
DPDP-inf
DPDP-ft
halfcheetah-medium-v2
∞
50.8
50.9
2
50.9
50.7
1
50.9
51.9
halfcheetah-medium-replay-v2
∞
46.0
46.2
Appendix
Table A5: Value-strength β sweep on the HalfCheetah tasks. β=∞ zeros the value branch and recovers the base planner, while β=1 is full value transfer. The default operating point used in the main results is β=1 .
β
DPDP-inf
DPDP-ft
hopper-medium-v2
∞
82.3
83.2
2
83.9
83.3
1
84.3
83.2
hopper-medium-replay-v2
∞
91.2
91.1
Appendix
Table A6: Value-strength β sweep on the Hopper tasks. Conventions as in Table A5 .
β
DPDP-inf
DPDP-ft
walker2d-medium-v2
∞
83.3
83.4
2
83.3
84.8
1
85.5
85.0
walker2d-medium-replay-v2
∞
85.7
85.6
Appendix
Table A7: Value-strength β sweep on the Walker2d tasks. Conventions as in Table A5 .
Task
M=0
M=1
M=3
M=5
halfcheetah-medium-v2
50.8
50.9
50.8
50.8
halfcheetah-medium-replay-v2
46.4
46.8
46.8
46.4
halfcheetah-medium-expert-v2
94.0
94.6
94.8
92.9
hopper-medium-v2
83.7
84.3
84.5
84.4
hopper-medium-replay-v2
92.2
92.2
91.2
91.5
hopper-medium-expert-v2
110.3
110.7
110.9
109.9
Appendix
Table A8: DPDP-inf as a function of the Langevin corrector budget M (normalized score; M=0 is the uncorrected composition). Suite-average rows are labeled in bold ; the best budget in each row is bold .
Task
M=0
M=1
M=3
M=5
marginal
DPDP-ft
halfcheetah-medium-v2
7.7
8.6
9.0
10.3
6.0
4.5
halfcheetah-medium-replay-v2
7.7
7.2
8.6
13.6
5.8
2.4
halfcheetah-medium-expert-v2
6.8
9.4
10.0
10.4
6.0
3.3
hopper-medium-v2
7.9
7.2
8.7
12.7
4.0
3.0
hopper-medium-replay-v2
6.4
7.0
10.7
11.7
4.4
2.0
hopper-medium-expert-v2
7.9
7.4
8.9
10.3
5.9
3.3
Appendix
Table A9: Wall-clock per evaluation run (minutes) on locomotion. M=0 – 5 are DPDP-inf with that many Langevin corrections; marginal is the plain single-model sampler; DPDP-ft is the amortized student. Lowest per row in bold .