cs.LGNov 1, 2026

Iterative Gumbel Planning for Continuous Control

Authors: Shaohuai Liu, Weirui Ye, Yilun Du, Le Xie

Organizations: Texas A&M University · CSAIL, MIT · Harvard University

Abstract

Search-based planning remains a powerful policy improvement paradigm in reinforcement learning, driving milestone successes in discrete domains. However, extending this paradigm to continuous control exposes a structural mismatch: search evaluates a finite set of action candidates, whereas continuous RL typically relies on Gaussian policies. We show that winner-action distillation into a continuous Gaussian induces a projection error and a stepwise variance-expanding gradient tendency when the selected action is sufficiently far from the current mean. Empirically, this tendency is associated with elevated terminal variance and a return gap on the evaluated tasks. To address this, we propose Iterative Gumbel Planning (IGP). By adopting a per-dimension factorized categorical policy, IGP naturally aligns the policy support with the search candidates, eliminating the projection error and reducing the residual approximation to a bounded discretization error. To bypass the combinatorial explosion of high-dimensional discrete action spaces, IGP pairs Gumbel Top-KK candidate sampling with an MPPI-style multi-round iterative search to refine the policy at each state. Empirically, IGP achieves state-of-the-art aggregate performance and sample efficiency on DMControl and HumanoidBench. Furthermore, it produces more stable action sequences with lower temporal variance, demonstrating its practical potential for real-world robotic deployment.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. GEM-MPC: Balancing Exploration and Exploitation through Expert-Guided Planning

    Sep 18, 2026Alvaro Serra-Gomez, Thomas MoerlandRL ExplorationModel-Based Planning

  2. In-Context Planning with Latent Temporal Abstractions

    Feb 21, 2026Baiting Luo, Yunuo Zhang, Nathaniel S. Keplinger +3Model-Based PlanningOffline RL

  3. Long-Horizon Multi-Object Rearrangement via Distilled Tree Search over Continuous Actions

    Oct 19, 2026Kallol Saha, Yining Hong, Amber Li +3Long-Horizon Robotic ManipulationModel-Based Planning