Teaching a Tree to Recover: Online Distillation of an LLM Supervisor with Counterfactual Replay

Kristian Ceder; Kilian Freitag; Knut Åkesson

cs.ROOct 29, 2026

Teaching a Tree to Recover: Online Distillation of an LLM Supervisor with Counterfactual Replay

Authors: Kristian Ceder, Kilian Freitag, Knut Åkesson

Organizations: Department of Electrical Engineering, Chalmers University of Technology, Sweden

Abstract

Autonomous mobile robots commonly pair a global planner with a local model predictive controller (MPC), but nonconvex obstacle avoidance and a limited horizon can make the robot freeze in complex layouts. Deciding whether to replan, wait, or call for help is a supervisory decision. Recent work queries a large language model (LLM) supervisor on every anomaly, which is slow and costly. We instead distill the LLM online into a decision tree. After each training episode, counterfactual replay labels recoveries as needed, unnecessary, or harmful. These labels retire unhelpful rules. In simulation, on 20 unseen layouts where the MPC alone freezes, the tree handles at least 18 correctly across all seeds, either by reaching the goal or by calling for help when no feasible path exists. The tree matches both an always-query LLM supervisor and expert-designed rules, while issuing no LLM queries at test time. Handwritten rules cannot anticipate the long tail of situations a physical robot will encounter. Online distillation is a promising remedy: the tree handles familiar cases, and the LLM is consulted only for new ones, whose validated recoveries become new rules. Language models can thus also serve as designers of supervisors rather than only as runtime supervisors.

Figures & tables

Figure 1: An LLM teaches a decision tree to recover an MPC-driven robot, then leaves the loop. Left: the tree answers what it has seen, the teacher the rest. Right: the experience memory, built by counterfactual replay: the three recoveries of one learning layout, each re-run without itself and labeled needed, unnecessary (no difference) or harmful (cost time); the labels retire rules that only look right and are shown to the teacher in context. Bottom: 200 learning layouts, three runs.

Figure 2: The supervision loop. Top: the control loop at

5\text{\,}\mathrm{H}\mathrm{z}

, the same in every arrangement. Middle: every

2\text{\,}\mathrm{s}

the tree answers the report if a trusted rule matches and otherwise asks the model, whose answer becomes a rule. Bottom: between episodes each recovery is replayed with continue in its place and the label is filed on the rule that ordered it; retirement reads that record, and so does the teacher. A tree hit takes about

6\text{\,}\mathrm{\SIUnitSymbolMicro s}

, a model call a median of

1.5\text{\,}\mathrm{s}

Arrangement	Right /20	T / W / C	Steps	Calls / ep	Age	Tree
No recovery	0	14 / 6 / 0	–	–	–	–
Scripted rules	18	2 / 0 / 0	288	–	1.0	–
LLM alone	$16.3\pm 1.5$	3.3 / 0 / 0.3	$259\pm 20$	$24.6\pm 3.3$	$7.8\pm 1.0$	–
Frozen tree	$18.3\pm 0.6$	1.7 / 0 / 0	$260\pm 5$	0	1.0	99%

Table 1: Held-out layout mean over three runs (

\pm

s.d.). Right : layouts ending right, of 20 (the goal, or a human called where one was needed); T / W / C : timeouts / watchdog stops / collisions; steps : to the goal; age : control cycles from report to verdict; tree : ticks answered by a rule. Frozen tree: the tree each method run ended with after the 200 learning layouts, run with no model.

Arm	Right /200	T / W / C	Recov.	Waits	Needed	Rules (ret.)	Unseen /100
Scripted rules	197	2 / 0 / 1	353	148	67%	–	95
Distillation alone	$192\pm 0$	7 / 0 / 1	$673\pm 0$	$517\pm 0$	37%	$8\pm 0$ (0)	$92\pm 0$
Without replay	$194.0\pm 1.7$	5.0 / 0 / 1	$679\pm 5$	$505\pm 11$	–	$15\pm 6$ (0)	$92.7\pm 0.6$
Method	$\mathbf{198.0\pm 1.0}$	2.0 / 0 / 0	$552\pm 46$	$351\pm 31$	41%	$30\pm 5$ (3)	$\mathbf{94.7\pm 2.3}$

Table 2: The 200 learning layouts, the same seeds for every arm; means over three runs,

\pm

s.d. (the scripted rules were run once). T / W / C : timeouts / watchdog stops / collisions; recov. : recoveries ordered, of which waits ; needed : share the replay labeled needed, where it runs; rules (ret.) : final rules (retired by replay labels); unseen : the tree each run ended with, run frozen on the 100 unseen layouts, right of 100 (rules: on the same layouts).

Figure 3: The 200 learning layouts in sequence: thin lines are runs, thick their mean. (a) Every 25 episodes, the tree is saved and run frozen, with no model, on the 20 held-out layouts: layouts ending right, of 20 (dotted: the scripted rules’ 18). (b) Teacher calls per episode, a 3-episode moving average on a logarithmic episode axis.

Figure 4: Self-improvement on a repeated layout. For each of eight layouts, the method starts from an empty tree and drives the same layout six times in a row; the tree it has after the sixth is then frozen, with no model, and drives it once more (F). Three independent repeats per layout, each from an empty tree; thin lines are layouts (mean of their repeats), thick lines are their mean. (a) Teacher calls per episode. (b) Ticks answered by a rule. (c) Cycles from report to verdict. (d) Runs ending right, of the 24 (eight layouts, three repeats).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The scenario grid, one staged situation per panel, from late in the method’s first learning run, where the tree answers every tick. Dark: the $3{\times}3$ blocks; orange: a staged obstacle the planned route (dashed) runs through, which the planner was not told about; gray: staged walls the planner knows, which form a gate or close the other way round. The driven path is shaded by time; red: pedestrian paths, and each pedestrian’s position at the first recovery. (a) A wall: the tree replans at the first tick, before the robot has stopped, and the new route (dotted) goes around the block. (b) A gate: the robot holds while a pedestrian stands in it, until the pedestrian has left. (c) A sealed passage: the replan finds no route, and the tree calls a human. (d) A breakdown: the blocker never leaves; the hold is let go once it has stood $30\text{\,}\mathrm{s}$ , and the tree calls a human.

					Harmful
Arm	Replayed	Clean	Needed	Unnec.	ending	steps	Unclear
Scripted rules	351	94%	67%	26%	0%	6%	1%
Distillation alone	$571\pm 6$	98%	$37\pm 1$ %	$10\pm 0$ %	$6\pm 0$ %	$46\pm 0$ %	1%
Method	$528\pm 29$	99%	$41\pm 1$ %	$17\pm 3$ %	$1\pm 1$ %	$41\pm 4$ %	0%

Appendix

Table 3: Counterfactual labels of the recoveries on the learning streams (Sec. 4.5 ), each replaced by continue in a replay run by the loop itself (at most six per episode); means

\pm

s.d. over three runs. Clean : replays exact up to the recovery’s own step, of which the shares are taken. Harmful : the replay flipped the ending to right, or only arrived more than 10 steps sooner. Experience without replay runs no replay. The scripted rules’ single run was replayed after the fact with the same settings (160 of 200 episodes exact; 94% of recoveries clean to their own step).

Robot Failure Recovery

Robot failure recovery studies how robots detect, diagnose, and respond to errors, damage, and unexpected physical interactions.

See all 74 papers

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL
Aug 25, 2026Xiaolong Jin, Dingmin Wang, Vijay Lingam +1RL for LLM Agents LLM-Guided RL
AWS AI Labs · Purdue University
BT-Agent: Deploying Onboard Robot Agents with Behavior Trees and Multi-Fidelity Evaluation
Oct 9, 2026Fabio Hübel, Adam Lilja, Yue Yin +5Large Language Model-Based Robot Planning Mobile Robotics
Stanford University · University of California, Berkeley · NVIDIA
A Tutorial on Autonomous Fault-Tolerant Control Using Knowledge-Grounded LLM Agents
Jun 30, 2026Javal Vyas, Milapji Singh Gill, Artan Markaj +2Language Model-Based Control Fault-Tolerant Control
Autonomous Industrial Systems Laboratory, Imperial College London, London, United Kingdom · Institute of Automation Technology, Helmut Schmidt University, Hamburg, Germany

Aug 25, 2026J/K move · Enter open · S save

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

Xiaolong Jin, Dingmin Wang, Vijay Lingam, Varun Kumar

AWS AI Labs · Purdue University

RL for LLM Agents LLM-Guided RL Multi-Turn RL Step-Level Credit Assignment in RL Self-Distillation

Open paper AskPDF arXiv

Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.

Abstract

Robot Failure Recovery

Explore similar work

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

BT-Agent: Deploying Onboard Robot Agents with Behavior Trees and Multi-Fidelity Evaluation

A Tutorial on Autonomous Fault-Tolerant Control Using Knowledge-Grounded LLM Agents