Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs

Yining Hong; Huang Huang; Manling Li; Li Fei-Fei; Leonidas Guibas; Jiajun Wu; Yejin Choi

cs.LGFeb 24, 2026

Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs

Authors: Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Leonidas Guibas, Jiajun Wu, Yejin Choi

Organizations: Stanford University · Northwestern University

Abstract

Embodied LLMs endow robots with high-level task reasoning, but they cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials where mistakes repeat rather than accumulate into experience. Drawing upon human reflective practitioners, we introduce Reflective Test-Time Planning, which integrates two modes of reflection: reflection-in-action, where the agent uses test-time scaling to generate and score multiple candidate actions using internal reflections before execution; and reflection-on-action, which uses test-time training to update both its internal reflection model and its action policy based on external reflections after execution. We also include retrospective reflection, allowing the agent to re-evaluate earlier decisions and perform model updates with hindsight for proper long-horizon credit assignment. Experiments on our newly-designed Long-Horizon Household benchmark and MuJoCo Cupboard Fitting benchmark show significant gains over baseline models, with zero-shot generalization to photorealistic HM3D environments and real-robot experiments on a Franka Panda arm. Ablations confirm that reflection-in-action and reflection-on-action are mutually dependent, and that retrospective reflection achieves better credit assignment than step-wise external feedback at lower computational overhead. Qualitative analyses further highlight behavioral correction through reflection.

Figures & tables

Figure 1: Conceptual overview of Reflective Test-Time Planning. The agent (a) receives a long-horizon task, (b) performs reflection-in-action by internally simulating and scoring candidate actions, (c) performs reflection-on-action by updating its beliefs and decision systems based on execution outcomes, and (d) conducts retrospective reflection to revise earlier decisions with hindsight.

Figure 2: Method overview. (a) Reflection-in-action : multiple candidate actions are generated and scored by an internal reflection LLM prior to execution. (b) Reflection-on-action : iteratively invoked when working memory hits K or at key milestones. Executed actions are critiqued by an external reflection LLM and stored in a working memory buffer; at milestones, hindsight re-evaluation assigns long-horizon credit. The resulting verbal reflections form self-supervised training data to update both the internal reflection LLM (supervised loss) and the action LLM (policy gradient) via test-time training, enabling agents to learn from execution experience during deployment.

	Baselines						Reflective Test-Time Planning Ablations
	Reflexion	Self-Refine	ReflectVLM	PPO	DreamerV3	3DLLM-Mem	w/o RIA/ROA	w/o RIA	w/o ROA	w/o Act.	w/o Int.	Ours
Fitting	8.51%	10.6%	2.12%	0%	4.26%	10.6%	0%	33.5%	6.38%	25.5%	12.8%	44.7%
Selection	8.82%	11.8%	5.88%	2.94%	11.8%	14.7%	17.6%	5.88%	11.8%	26.5%	8.82%	32.4%
Preparation	15.9%	12.7%	14.3%	7.94%	11.1%	9.52%	11.1%	3.17%	19.0%	20.6%	17.5%	31.7%
Hybrid	6.45%	9.68%	6.45%	3.23%	12.9%	9.68%	12.9%	3.23%	12.9%	16.1%	9.68%	25.8%
Average	9.92%	11.20%	7.19%	3.53%	10.02%	11.13%	10.40%	11.45%	12.52%	22.18%	12.20%	33.65%

Table 1: Baseline comparisons and ablations of Reflective Test-Time Planning on long-horizon household tasks. RIA denotes reflection-in-action and ROA denotes reflection-on-action. Act. denotes action model loss; Int. denotes internal reflection model loss.

Table 4

Figure 3: Cupboard Fitting results . Blue bars show correct-placement rate, pink bars show fit rate. RIA means Reflection-in-action; ROA means Reflection-on-action. “W/o external reflection”: we don’t use external reflection as the input to the action generation LLM. We implement two test-time training variants for ROA: 1) test-time training on all base weights; 2) test-time training on LoRA parameters only. Reflective Test-Time Planning significantly improves both success metrics.

Figure 5: Qualitative Examples. Steps and reflections simplified for better presentations. Blue text: internal reflection. Orange text: external reflection. red text: retrospective reflection. (a) Long-Horizon Household example. We use retro & internal because the generated retro reflection is also used to train the internal model. (b) Real-robot Cupboard Fitting example. We put reflection scores inside brackets, omit detailed reflections and only present the scores for simplicity.

Table 8

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	w/o RIA w/o ROA		Ours (Full)
	Vanilla	Same Time Budget
Fitting	0.00%	0.00%	44.7%
Selection	17.6%	14.7%	32.4%
Preparation	11.1%	12.7%	31.7%
Hybrid	6.45%	6.45%	25.8%
Average	8.79%	8.46%	33.65%

Appendix

Table 6: Performance comparison between (1) vanilla ablation without RIA or ROA, (2) vanilla baseline with matched time budget (3× steps), and (3) our full reflective model on Long-Horizon Household Tasks.

Configuration	Success	Overhead
Ours (full)	33.7%	1.00 $\times$
+Parallel Scoring + KV Cache	33.0%	0.85 $\times$
+Batched Retro-Reflection	33.5%	0.66 $\times$
+TTT Early-Stop	32.3%	0.60 $\times$
+QLoRA	32.0%	0.52 $\times$
+Multi-GPU Parallelism	31.5%	0.45 $\times$

Appendix

Table 7: Cumulative efficiency improvements. Each row adds to the previous. The combined pipeline reduces overhead to

0.45\times

with minimal performance degradation.

Setting	Factual	Causal	Useful
In-domain (BEHAVIOR)	98	95	99
Out-of-domain (HM3D)	96	96	95

Appendix

Table 8: Human evaluation of

V_{\phi_{e}}

reflection quality (%). Quality remains high under distribution shift.

Method	HM3D	Cleanup	Rearrange
Ours w/ $V_{\phi_{e}}$	19.5%	40.0%	26.7%
Ours w/ Human Oracle	21.9%	40.0%	33.3%

Appendix

Table 9:

V_{\phi_{e}}

vs. human oracle under distribution shift. Near-identical performance confirms

V_{\phi_{e}}

provides supervision of equivalent quality to human judgment.

Method	Accuracy (%)
Zero-SFT, w/o RIA & ROA	0.0
Zero-SFT, w/ RIA & ROA	0.0
Zero-SFT, Few-Shot In-Context, w/o RIA & ROA	1.3
Zero-SFT, Few-Shot In-Context, w/ RIA & ROA	2.1
Ours, w/o RIA & ROA	10.4
Ours (Full)	33.7

Appendix

Table 10: Zero-SFT baselines confirm that test-time reflection cannot function without minimal SFT initialization.

Model	Sem. Sim.	Human Eval	Format Parse	Fit %
Qwen2-VL 2B (no SFT)	0.59	76	0.19	4.7
Qwen2.5-VL 3B (no SFT)	0.73	89	0.27	8.0
Qwen2.5-VL 7B (no SFT)	0.75	91	0.31	11.9
Qwen2-VL 2B + SFT	—	—	0.95	48.5
Qwen2.5-VL 3B + SFT	—	—	0.99	60.2
Qwen2.5-VL 7B + SFT	—	—	0.99	52.7

Appendix

Table 11: Base model scale analysis. Without SFT, 3B/7B models have sufficient semantic quality but near-zero format parse rates; SFT fixes format. Qwen2.5-VL 3B hits the optimal fit rate; 7B degrades due to over-parameterization.

Metric	BEHAVIOR	HM3D (OOD)
Revision Direction (%)	99	95
Human Eval — Factual	98	96
Human Eval — Causal	95	96
Human Eval — Useful	99	95

Appendix

Table 12: Retro-reflection quality. Score revision directions align with ground-truth outcomes at

\geq

95% even out-of-domain, and human evaluators rate reflections as highly factual, causally grounded, and useful.

Method	Fit	Sel.	Prep.	Hybrid	Avg.	Overhead
LG-MCTS	14.9	14.7	15.8	16.1	15.4	$\sim$ 5–8 $\times$
Ours w/o TTT	6.4	11.8	19.0	12.9	12.5	0.5 $\times$
Ours	44.7	32.4	31.7	25.8	33.7	1 $\times$

Appendix

Table 13: LG-MCTS is inferior to our full method at 5–8

\times

cost. Search-based lookahead is ungrounded in physical uncertainties that can only be resolved through interaction; TTT, not search depth, is the key.

Instruction Type	Fit	Sel.	Prep.	Hybrid	Avg.
GPT-Generated (current)	44.7	32.4	31.7	25.8	33.7
Human-Written	42.5	33.6	25.7	27.4	32.3

Appendix

Table 14: Performance generalizes to human-written instructions with marginal degradation.

Method	Fit	Sel.	Prep.	Hybrid	Avg.
GPT-5	19.2	23.5	17.4	22.6	20.7
Gemini 2.5 Pro	14.9	20.5	20.6	22.6	19.7
Qwen2.5	12.8	14.6	14.3	15.1	14.2
Ours	44.7	32.4	31.7	25.8	33.7

Appendix

Table 15: Commercial LLM zero-shot baselines. Our framework outperforms all commercial models despite using a 3B base model, demonstrating that test-time learning, not model scale, drives the gains. GPT-5 as a zero-shot baseline receives only partial observations; its data-generation role uses complete scene graphs.

Method	Consistent w/ Sim.	Consistent w/ Human
GPT Raw Scores	85%	80%
GPT + Sim. Validation	100%	98%

Appendix

Table 16: GPT-5 data quality after physics-simulation validation. Simulator filtering raises human consistency from 80% to 98%.

Dimension	Score
Task Naturalness	95
Reflection Quality	99
Score Calibration	97
Diversity	92

Appendix

Table 17: Human evaluation of GPT-5-generated data quality.

Figure 6: Hyperparameter ablation studies on Cupboard Fitting. Top Left: Performance vs. number of candidate actions. Peak performance (60.0%) occurs at N=6 candidates, demonstrating that internal reflection effectively identifies superior actions from diverse pools. Beyond N=6, performance plateaus as excessive candidates add computational cost without improving the best candidate quality. Top Right: Performance vs. sampling temperature. Optimal temperature range (T=1.25-1.5) balances candidate diversity with quality—temperatures below 0.5 produce overly similar candidates that limit reflection value, while temperatures above 1.75 generate incoherent actions that even accurate reflection cannot salvage. Bottom Left: Performance vs. LoRA configuration (rank, alpha). The optimal configuration (r=8, $\alpha$ =16) achieves 60.0% performance, balancing adaptation capacity with training stability. Smaller configurations like (4,4) underfit with insufficient capacity (52.5%), while larger configurations cause mode collapse during test-time training—(16,32) drops to 41.5% and (32,32) collapses to 34.8% as the model begins predicting identical outputs for all inputs, losing the ability to distinguish between different spatial configurations and task contexts. Bottom Right: Performance vs. action budget (maximum steps). Performance improves dramatically from 30 steps (51.5%) to 50 steps (60.0%), but slightly degrades to 59.4% at 100 steps, suggesting that excessive action budgets allow suboptimal exploration strategies that accumulate errors over longer horizons.

Method	Fit	Correct	Compute
Ours (Receding Horizon)	57.8%	25.8%	5.0×
Ours (Single-Step)	60.2%	25.3%	1.0×

Appendix

Table 18: Single-step action generation vs. receding horizon planning on Cupboard Fitting. Despite using 5× more computation to plan sequences at each step, receding horizon shows degraded performance, demonstrating that single-step action generation is more effective for our reflective learning framework.

Figure 7: Distribution of task categories in the dataset.

Figure 8: Additional qualitative examples. Blue text: internal reflection for candidate selection. Orange text: external reflection after execution. Red text: retrospective reflection and model updates. Reflection scores shown in brackets. Steps and reflections simplified for presentation.

Large Language Model-Based Robot Planning

Large language model-based robot planning turns natural-language goals and sensory observations into grounded, executable actions for robots.

See all 219 papers

Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL
Jun 12, 2026Yinglun ZhuLLM Agent Evaluation Language Model Calibration
University of California, Riverside
Spotter: Let the Embodied Model Lead, and the VLM Reflect for It
Sep 29, 2026Long Li, Qichao Zhao, Yue Yang +6Robot Failure Recovery Vision-Language-Action Models
Griffith University · IIIS, Tsinghua University · MainCode +3
Agent-Reflex: Teaching Language Agents to Act Reflectively
Oct 13, 2026Guanting Dong, Xinyu Yang, Jiajie Jin +9LLM Agent Training Self-Reflection in Language Model Agents
Gaoling School of Artificial Intelligence, Renmin University of China · National University of Singapore

Jun 12, 2026J/K move · Enter open · S save

Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL

Yinglun Zhu

University of California, Riverside

LLM Agent Evaluation Language Model Calibration Self-Improving Agents RL for LLM Agents Self-Reflection in Language Model Agents

Open paper AskPDF arXiv

LLMs are increasingly deployed as agents that interact with external environments and observe feedback such as execution results, error messages, and tool outputs. A well-functioning agent should be able to leverage this feedback to accurately assess its own performance. Yet we find a persistent reflection gap: LLM agents tend to mis-assess their own outputs after observing concrete environment feedback -- even for questions they correctly answered -- and standard RL barely helps due to a credit-assignment mismatch. To close this gap, we propose RefGRPO, a simple yet effective fix that augments standard RL algorithms with two key ingredients: a free calibration bonus computed by contrasting the agent's own reflection with the actual outcome (requiring no additional reward model, LLM judge, or external annotation), and a dynamic schedule on its coefficient. Compared to standard RL baselines, our method simultaneously improves reflection calibration (e.g., reduces underconfidence rate

44.4\% \to 7.7\%

) and task accuracy (e.g.,

75.1\% \to 76.5\%

) on text-to-SQL across five benchmarks. The resulting calibrated reflection turns the agent into its own verifier grounded in environment feedback, which further enables (i) better self-improvement that uses reflections as pseudo-rewards without outcome supervision, and (ii) more effective test-time selective prediction by committing only to rollouts flagged as correct.

Abstract

Large Language Model-Based Robot Planning

Explore similar work

Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL

Spotter: Let the Embodied Model Lead, and the VLM Reflect for It

Agent-Reflex: Teaching Language Agents to Act Reflectively