Online decision-making faces a tension between efficient inference and long-horizon reasoning. Value-based methods rely on short-horizon bootstrapping that limits the propagation of long-term information, while trajectory optimization methods plan explicitly but must solve a high-dimensional optimization problem at every decision step. We propose Generative Trajectory Planning (GTP), a model-based framework that addresses this tension by learning a generative model as a proposal distribution over action sequences. Instead of optimizing sequences from scratch, GTP samples candidate sequences from a diffusion prior and refines them using learned dynamics, reward, and value models, enabling efficient trajectory-level reasoning. Across continuous control and long-horizon manipulation tasks, GTP matches or exceeds state-of-the-art model-free, diffusion-policy, and model-based baselines, and remains stable in regimes where strong baselines degrade. Our results suggest that generative models are best understood not as policies, but as structured proposal distributions for planning, offering a new perspective on how to integrate generative modeling with model-based decision-making.
Figures & tables
Figure 1: Generative Trajectory Planning. GTP combines the data-driven nature of RL with the long-horizon reasoning of trajectory optimization. A learned diffusion prior proposes candidate action sequences, which are evaluated and iteratively refined through learned models. By using a generative prior, GTP avoids computational cost of optimization while preserving long-horizon reasoning.
Figure 2: Generative Trajectory Planning (GTP). Planning proceeds via iterative beam search. Starting from an initial beam, GTP repeatedly (1) branches candidate trajectory prefixes by duplicating, (2) refines action suffixes through partial forward diffusion and denoising under the diffusion prior, and (3) prunes candidates using rollouts through learned dynamics, reward, and value models. The top-scoring trajectories are retained and expanded at the next depth, progressively constructing long-horizon plans toward high-return regions.
Figure 3: GTP consistently outperforms baselines. Evaluation return curves on OpenAI Gym MuJoCo environments. Solid lines show the mean and shaded regions the standard error over 5 random seeds.
Algorithm
Hopper-v4
Walker2d-v4
HalfCheetah-v4
Ant-v4
Humanoid-v4
Classic
SAC
3351 ± 39
4453 ± 158
10515 ± 286
4229 ± 366
5118 ± 153
Model-Free RL
TD3
3314 ± 47
4408 ± 141
9301 ± 546
4278 ± 293
3852 ± 967
Diffusion
QSM
1146 ± 133
839 ± 212
9084 ± 179
1868 ± 97
1294 ± 207
Policy RL
DPMD
3147 ± 142
4133 ± 109
11591 ± 123
4092 ± 354
5269 ± 44
Trajectory
CEM-MPC
141 ± 37
71 ± 31
651 ± 60
379 ± 53
318 ± 17
Optimization
CEM-MPC-V
2437 ± 345
288 ± 54
7293 ± 211
5103 ± 811
4396 ± 569
Table 1: Results on locomotion benchmark. Performance on OpenAI Gym MuJoCo environments. We report mean returns and standard errors over 5 random seeds, each trained for 1M environment interactions.
Figure 4: GTP improves sample efficiency in long-horizon settings. Evaluation return curves on Fetch and PushT environments. Solid lines show the mean and shaded regions the standard error over 5 random seeds.
Algorithm
FetchPush
FetchPick&Place
FetchSlide
PushT-Vision
Classic
SAC
6 ± 0
3 ± 0
1 ± 1
0 ± 0
Model-Free RL
TD3
6 ± 0
3 ± 0
5 ± 1
0 ± 0
Diffusion
QSM
12 ± 3
4 ± 0
14 ± 2
0 ± 0
Policy RL
DPMD
6 ± 0
3 ± 0
5 ± 1
0 ± 0
Trajectory
CEM-MPC
40 ± 4
33 ± 5
50 ± 3
16 ± 2
Optimization
CEM-MPC-V
78 ± 6
68 ± 5
67 ± 1
33 ± 4
Table 2: Long-horizon manipulation results. Success rates (%) on long-horizon manipulation tasks. Mean and standard error over 5 random seeds, each trained for 1M environment interactions.
Figure 5: Ablations of key components. Best mean returns and standard errors over 5 random seeds.
Figure 8
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
MuJoCo
Manipulation
Diffusion Policy (UNet)
Planning horizon H
4
8
Base channel dim
64
32
Channel multipliers
[1,2]
[1,2,4]
Conditioning embedding dim
64
32
Dynamics Ensemble
Appendix
Table 3: Hyperparameters for GTP across task suites.
Figure 8: Effect of ensemble aggregation on Ant-v4. Pessimistic aggregation (min/median for update/rollout) substantially outperforms optimistic aggregation (median/max) on this termination-sensitive environment.
beam width \ num. beams
1
2
4
8
2
13944 ± 498
13830 ± 517
13338 ± 174
12892 ± 778
4
11819 ± 481
13587 ± 572
12614 ± 792
12268 ± 326
Appendix
Table 4: HalfCheetah-v4 environment best mean returns and standard errors over 200k iterations with 5 random seeds for different beam widths B and number of beams.
SAC
TD3
QSM
DPMD
DreamerV3
TD-MPC2
GTP (ours)
Success Rate
0 ± 0
0 ± 0
0 ± 0
0 ± 0
0 ± 0
97 ± 1
99 ± 1
Appendix
Table 5: PushT-Vision success rate (%) at 3M environment interactions. Mean and standard error over 5 seeds.
Figure 9: Left: Rendering of the PushT-Vision environment. Right: Evaluation returns on PushT-Vision over 3M environment interactions. Solid lines show the mean and shaded regions the standard error over 5 seeds.
Method
Action selection (ms)
Update (ms)
SAC
0.06
0.78
DPMD
0.73
2.18
TD-MPC2 ( H=3 )
17.19
2.44
TD-MPC2 ( H=8 )
36.20
3.99
GTP ( H=1 )
1.02
8.54
GTP ( H=4 )
21.20
82.83
Appendix
Table 6: Computational cost of GTP and representative baselines. Per-step action selection and training update times (ms) under matched hardware and batch settings.
Figure 10: Trajectory reuse reduces training overhead while preserving performance. Reusing elite trajectories maintains comparable learning curves (top) and final success rates (bottom).