Planning while moving faces an historical trade-off between deliberation and reactivity. A planner running concurrently with execution needs time to compute, so the robot must commit to an open-loop action chunk for that duration: a longer commitment buys a better plan at the cost of feedback frequency. Existing controllers fix this action commitment, and consequently the computation budget, before deployment. However, the optimal computation time is highly state-dependent (due to factors such as unmodeled dynamics, safety constraints, and marginal optimisation gains), i.e. dynamic, therefore offline static tuning risks leaving systems blind during critical maneuvers or unable to find global effective solutions. We cast the identification of the optimal computation budget as a semi-Markov decision process (SMDP) in which a single decision defines both the robot action commitments and the time allocated to the solver. To solve this, we train a reinforcement-learning policy conditioned on the current state, the executing plan, and inexpensive planner statistics. Across ground-robot navigation, agile quadrotor flight, and contact-rich manipulation, the learned policy changes its budget with the demands of the scenario, discovering efficient reasoning patterns that outperform fixed-budget baselines. https://lorenzomagnino.github.io/learning-robot-metareasoning-review/
Figures & tables
Fig. 1 : Robot Metareasoning: different states require different computation budgets . Robots must adapt their computation budget based on the situation that they are facing: ( s ) deliberate longer to search for an optimal path, ( s′ ) react rapidly to noisy moving obstacles, or ( s′′ ) balance planning and reactivity. Across real-world and simulated experiments, we show that adapting the computation budget across states within an episode is a fundamental capability for embodied intelligence.
Fig. 2 : Method Overview . At each decision epoch, the state, the previous plan and its statistics inform the budget policy Φθ , which selects the next execution horizon and thereby allocates the solver’s budget. This lets the robot decide how much to deliberate under environmental uncertainty and temporal constraints. The new plan takes over, the state and statistics are refreshed, and the loop repeats.
Fig. 3 : Approaches . (Left) Real-time MPC commits to one step ( h=1 ), whereas action chunking (right) executes a fixed block of h actions.
Fig. 4 : Navigation Results , pooled over the three scenarios ( 300 randomised episodes). Our learned budget policy (left of the dashed rule) against every fixed commitment h (blue). The two right-hand panels condition on successful episodes and therefore flatter the low-success baselines. The solid and dashed outlines mark the best fixed budget under success rate ( h=5 ) and under total return ( h=1 ): no single fixed h is best at both, while the learned budget dominates both.
Fig. 5 : Real-world Merging Lane scenario. Top: temporal composites on the RoboMaster platform under our learned budget and fixed commitments h∈{1,4,9} ; each panel overlays successive robot poses from one episode. Bottom: the commitment our policy selects along its episode, against the three fixed budgets. The budget collapses to h=1 while the robot merges past the crossing obstacle (red band) and while it threads the close passage between the static obstacles.
Fig. 6 : Optimality Regret in the Merging Lane scenario along three fixed commitment trajectories. Lines are means and standard deviations over 10 episodes. The tint marks Phase 1, before the moving obstacle has been passed, which ends at a different step for each commitment. Dotted rules mark the mean collision step for each fixed strategy.
Fig. 7 : Holonomic 2D navigation: planner statistics ablation . Ablation of the policy input, pooled over the three scenarios ( 300 episodes per variant). Full uses (stk,hk−1,τk,bk) ; PS only uses the planner statistics bk alone; Full − PS uses everything except bk . The full input achieves the highest success rate and total return, while removing planner statistics drops success by 12 percentage points.
Fig. 8 : Dynamic commitment during cable-payload flight. Four stages from one episode show the learned policy increasing the commitment for transit and reducing it near each target.
Fig. 9 : Cable-payload slalom return. Mean total return over 20 episodes per method. The left group compares the full policy (gold) with ablations without planner statistics (orange) and using planner statistics only (salmon); fixed commitments are shown in blue.
Scenario
Strategy
Success ↑
Total return ↑
Harshness ↓
Barrier [1pt]
Ours
0.97
−435
7.37
h=5 (S)
0.73
−1167
7.28
h=1 (R)
0.12
−1523
7.42
h=9 (H)
0.63
−1400
7.55
Merging Lane [1pt]
Ours
1.00
−385
7.02
h=5 (S)
0.63
−1462
7.17
TABLE I : Navigation results ( 100 episodes per scenario). We compare our policy with the same three fixed budgets selected from the pooled results in Fig. 4 : h=5 (best Success rate), h=1 (best total Return), and h=9 (best Harshness). The budgets remain fixed across scenarios; total return is averaged over all episodes.
Fig. 10 : Boundary behaviour on the 3D PushT environment . Left: robot-arm manipulation setup. Right: terminal position and orientation errors after 600 steps, reported as mean ± sample standard deviation over 16 matched validation episodes. Our policy selects hˉ=8.00 .
It has long been recognized that humans have the ability to switch between fast, reactive decision-making and slower, deliberative planning. In this paper, we study the question of how to learn this ability, known as meta-reasoning, in artificial agents. We model reactive decision-making as a policy that directly maps state observations to actions. Such policies can be trained with reinforcement learning (RL) or imitation learning, but may generalize poorly outside of their training distribution. Alternatively, model-based decision-time planning is more likely to produce good actions across a broader set of states but requires additional computation time, which delays acting. In this work, we introduce an RL method for training a meta-reasoning policy that allocates computation by conditioning on a reactive-policy uncertainty score. This score enables it to predict when the reactive policy is likely to perform poorly and when planning is needed. We conduct an empirical study on motion planning and navigation environments, showing that this design enables the meta-reasoning policy to learn when the reactive policy provides a good-enough action versus when decision-time planning is needed. Additionally, we show that our design enables the meta-agent to shift toward fully reactive control as the reactive policy improves.
Adam Labiosa, Josiah P. Hanna
Department of Computer Sciences, University of Wisconsin–Madison, USA
Robots operating in non-stationary environments must continually adapt their policies as the dynamics drift, but onboard energy and compute budgets cap how often a full state estimation and re-planning step can be performed. This raises a question: when, along a horizon, should a robot spend its limited budget? We formulate this problem in time-varying Markov decision processes (TVMDPs) with a known bound on the rate of transition drift. We model execution as a skip-update scheme in which, at chosen update times, the agent estimates the transition kernel by maximum likelihood and computes a finite-horizon policy, and between updates reuses this policy under a propagated state estimate. We analyze the dynamic regret of this scheme and show how it grows during skip intervals in terms of the properties of the TVMDP and the skip lengths; the resulting bound answers the opening question via an online, regret-guided update rule that allocates the budget adaptively. We evaluate the rule in a simulated Mars-rover navigation task with time-varying slip dynamics and on a Crazyflie quadrotor in indoor obstacle fields. Adaptive allocation outperforms other budgeted baselines.
Deliberating takes time. In real-time settings, that time is not free. Standard reinforcement learning (RL) sidesteps this as the environment waits indefinitely for the agent's decision. Instead, we study real-time RL environments where the environment progresses while waiting for the agent's action. Building on prior real-time formalizations, we introduce variable-delay real-time RL, where the agent chooses how long to deliberate at each decision point since the environment progresses. For the planning agents we use, the right delay is state-dependent, and naively planning how long to plan can paralyze the agent. We instead approach this setting by training a lightweight gating policy on top of a planner to select state-dependent planning budgets. Across real-time Pac-Man, Tetris, Snake, Speed Hex, and Speed Go, our gating policy outperforms fixed-budget and heuristic baselines, and transfers to a real-time setup where the environment and agent run on two different GPUs.
Aneesh Muppidi, Firas Darwish, Dylan Cope +2
British Open-ended Learning and Discovery Lab (BOLD), University of Oxford · Department of Statistics, University of Oxford · VGG, University of Oxford