Online decision-making faces a tension between efficient inference and long-horizon reasoning. Value-based methods rely on short-horizon bootstrapping that limits the propagation of long-term information, while trajectory optimization methods plan explicitly but must solve a high-dimensional optimization problem at every decision step. We propose Generative Trajectory Planning (GTP), a model-based framework that addresses this tension by learning a generative model as a proposal distribution over action sequences. Instead of optimizing sequences from scratch, GTP samples candidate sequences from a diffusion prior and refines them using learned dynamics, reward, and value models, enabling efficient trajectory-level reasoning. Across continuous control and long-horizon manipulation tasks, GTP matches or exceeds state-of-the-art model-free, diffusion-policy, and model-based baselines, and remains stable in regimes where strong baselines degrade. Our results suggest that generative models are best understood not as policies, but as structured proposal distributions for planning, offering a new perspective on how to integrate generative modeling with model-based decision-making.
Figures & tables
Figure 1: Generative Trajectory Planning. GTP combines the data-driven nature of RL with the long-horizon reasoning of trajectory optimization. A learned diffusion prior proposes candidate action sequences, which are evaluated and iteratively refined through learned models. By using a generative prior, GTP avoids computational cost of optimization while preserving long-horizon reasoning.
Figure 2: Generative Trajectory Planning (GTP). Planning proceeds via iterative beam search. Starting from an initial beam, GTP repeatedly (1) branches candidate trajectory prefixes by duplicating, (2) refines action suffixes through partial forward diffusion and denoising under the diffusion prior, and (3) prunes candidates using rollouts through learned dynamics, reward, and value models. The top-scoring trajectories are retained and expanded at the next depth, progressively constructing long-horizon plans toward high-return regions.
Figure 3: GTP consistently outperforms baselines. Evaluation return curves on OpenAI Gym MuJoCo environments. Solid lines show the mean and shaded regions the standard error over 5 random seeds.
Algorithm
Hopper-v4
Walker2d-v4
HalfCheetah-v4
Ant-v4
Humanoid-v4
Classic
SAC
3351 ± 39
4453 ± 158
10515 ± 286
4229 ± 366
5118 ± 153
Model-Free RL
TD3
3314 ± 47
4408 ± 141
9301 ± 546
4278 ± 293
3852 ± 967
Diffusion
QSM
1146 ± 133
839 ± 212
9084 ± 179
1868 ± 97
1294 ± 207
Policy RL
DPMD
3147 ± 142
4133 ± 109
11591 ± 123
4092 ± 354
5269 ± 44
Trajectory
CEM-MPC
141 ± 37
71 ± 31
651 ± 60
379 ± 53
318 ± 17
Optimization
CEM-MPC-V
2437 ± 345
288 ± 54
7293 ± 211
5103 ± 811
4396 ± 569
Table 1: Results on locomotion benchmark. Performance on OpenAI Gym MuJoCo environments. We report mean returns and standard errors over 5 random seeds, each trained for 1M environment interactions.
Figure 4: GTP improves sample efficiency in long-horizon settings. Evaluation return curves on Fetch and PushT environments. Solid lines show the mean and shaded regions the standard error over 5 random seeds.
Algorithm
FetchPush
FetchPick&Place
FetchSlide
PushT-Vision
Classic
SAC
6 ± 0
3 ± 0
1 ± 1
0 ± 0
Model-Free RL
TD3
6 ± 0
3 ± 0
5 ± 1
0 ± 0
Diffusion
QSM
12 ± 3
4 ± 0
14 ± 2
0 ± 0
Policy RL
DPMD
6 ± 0
3 ± 0
5 ± 1
0 ± 0
Trajectory
CEM-MPC
40 ± 4
33 ± 5
50 ± 3
16 ± 2
Optimization
CEM-MPC-V
78 ± 6
68 ± 5
67 ± 1
33 ± 4
Table 2: Long-horizon manipulation results. Success rates (%) on long-horizon manipulation tasks. Mean and standard error over 5 random seeds, each trained for 1M environment interactions.
Figure 5: Ablations of key components. Best mean returns and standard errors over 5 random seeds.
Figure 8
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
MuJoCo
Manipulation
Diffusion Policy (UNet)
Planning horizon H
4
8
Base channel dim
64
32
Channel multipliers
[1,2]
[1,2,4]
Conditioning embedding dim
64
32
Dynamics Ensemble
Appendix
Table 3: Hyperparameters for GTP across task suites.
Figure 8: Effect of ensemble aggregation on Ant-v4. Pessimistic aggregation (min/median for update/rollout) substantially outperforms optimistic aggregation (median/max) on this termination-sensitive environment.
beam width \ num. beams
1
2
4
8
2
13944 ± 498
13830 ± 517
13338 ± 174
12892 ± 778
4
11819 ± 481
13587 ± 572
12614 ± 792
12268 ± 326
Appendix
Table 4: HalfCheetah-v4 environment best mean returns and standard errors over 200k iterations with 5 random seeds for different beam widths B and number of beams.
SAC
TD3
QSM
DPMD
DreamerV3
TD-MPC2
GTP (ours)
Success Rate
0 ± 0
0 ± 0
0 ± 0
0 ± 0
0 ± 0
97 ± 1
99 ± 1
Appendix
Table 5: PushT-Vision success rate (%) at 3M environment interactions. Mean and standard error over 5 seeds.
Figure 9: Left: Rendering of the PushT-Vision environment. Right: Evaluation returns on PushT-Vision over 3M environment interactions. Solid lines show the mean and shaded regions the standard error over 5 seeds.
Method
Action selection (ms)
Update (ms)
SAC
0.06
0.78
DPMD
0.73
2.18
TD-MPC2 ( H=3 )
17.19
2.44
TD-MPC2 ( H=8 )
36.20
3.99
GTP ( H=1 )
1.02
8.54
GTP ( H=4 )
21.20
82.83
Appendix
Table 6: Computational cost of GTP and representative baselines. Per-step action selection and training update times (ms) under matched hardware and batch settings.
Figure 10: Trajectory reuse reduces training overhead while preserving performance. Reusing elite trajectories maintains comparable learning curves (top) and final success rates (bottom).
Diffusion-based trajectory planners have shown strong performance in offline reinforcement learning, but their iterative denoising process often incurs high inference cost. Consistency-based planners reduce the number of sampling steps, yet they typically rely on a two-stage teacher--student distillation pipeline that increases training cost and may introduce instability. We propose Shortcut Trajectory Planning (STP), an offline model-based reinforcement learning framework that incorporates shortcut models as efficient trajectory generators. STP trains a conditional shortcut trajectory model in a single stage, supports adjustable one-step and few-step inference through step-size conditioning, and selects candidate plans using a critic augmented with feasibility-aware correction. Across standard D4RL benchmarks, including locomotion, navigation, manipulation, and dexterous control tasks, STP achieves strong performance while simplifying the training pipeline for fast generative planning.
Guanquan Wang, Yoshimasa Tsuruoka
Department of Information and Communication Engineering · The University of Tokyo
Offline reinforcement learning (RL) leverages static datasets to learn decision policies without real-time environment interaction. While recent sequence-modeling approaches rely on continuous diffusion models for trajectory synthesis, applying these methods to discrete planning tasks requires a categorical formulation rather than the standard Gaussian construction. We present BFN-RL, a unified generative modeling framework for offline RL based on Bayesian Flow Networks (BFNs). By iteratively evolving distribution parameters rather than noisy data instances, BFN-RL natively models both discrete and continuous trajectory spaces within a single probabilistic formulation. The categorical planner generates future state sequences, and a learned inverse-dynamics model converts consecutive generated states into actions. Evaluations in discrete planning and continuous control show that BFN-RL can generate effective trajectories across both categorical and continuous state spaces. Our results establish BFNs as a versatile generative foundation for offline trajectory planning across data modalities.
Recent advances in generative planning have made trajectory inpainting a promising approach to offline goal-conditioned reinforcement learning. However, these methods typically specify the planning horizon before generating plan content, even though the appropriate horizon depends on the route itself. A horizon that is too short can force infeasible transitions, whereas one that is too long can introduce redundant motion. We introduce HorizonFlow, a hierarchical planner that treats plan length as an output of generation rather than a prescribed input. Its subgoal route planner guides its action-prefix controller through a sequence of latent subgoals. Both components combine insertion-based generation with flow matching to jointly generate continuous plan content and length, using the partially generated plan to guide token insertion. HorizonFlow reuses the resulting length information to select candidates and steer generation toward shorter plans without a separate learned value model. Across Maze2D, Multi2D, and OGBench navigation and visual manipulation benchmarks, HorizonFlow achieves the highest average performance among the compared methods.
JunHyeok Oh, Zian Jang, Byung-Jun Lee
Korea University. · Gauss Labs Inc., Seoul, Republic of Korea.