cs.ROOct 27, 2026

RoboAuto: Autonomous Robot Program Search through Multi-Level Physical Understanding

Authors: Buzhao Liu, Jianing Ye, Xiu Yuan, Yangtong Feng, Chengsong Huang, Tyler Dinh, Yevgeniy Vorobeychik, Ning Zhang, 

Organizations: Washington University in St. Louis

Abstract

Recently, LLM-based autonomous agents have shown strong capabilities in iteratively generating, evaluating, and refining executable solutions. In robotic manipulation, Code-as-Policy lets these agents search over modular and editable robot programs. However, effective program search requires physical understanding: agents must understand how objects can be manipulated, how scene geometry constrains actions, and why interactions succeed or fail. Existing methods rely mainly on execution feedback for program refinement, often leaving these physical questions unresolved. We introduce RoboAuto, a framework for robot program search that explicitly organizes physical understanding at the object, scene, and interaction levels through three components. First, a hierarchical knowledge base combines executable skills with object-centric knowledge of affordances, spatial relations, and action effects, learned from embodied data and online task solving. Second, scene grounding connects this knowledge to the current environment by identifying relevant objects and resolving action-critical quantities. Third, multi-stage planning with self-designed experiments checks whether each stage produces its expected physical effects and actively tests competing hypotheses to resolve physical uncertainties. RoboAuto achieves superior performance across five benchmark families, attaining an 89.1% success rate on short-horizon LIBERO-PRO tasks and more than twice the mean success rate of ASPIRE on LIBERO-PRO Long and LIBERO-Plus. It also outperforms ASPIRE and π0.5π_{0.5} on three challenging long-horizon real-robot tasks requiring precise manipulation. We further show that the knowledge base transfers across simulators and embodiments, and that program search performance scales with its size. Project page: https://roboauto-ai.github.io/

Figures & tables

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ASPIRE: Agentic /Skills Discovery for Robotics

    Jun 30, 2026Runyu Lu, Yubo Wu, Ethan Kou +13Robot Policy LearningLong-Horizon Robotic Manipulation

  2. RAPID: Robot Agentic Programming from Demonstrations

    Sep 24, 2026Yuyao Liu, Jiayuan Mao, David Hsu +2Program SynthesisContact-Rich Robotic Manipulation

  3. LIBERO-Agent: Evaluating General-Purpose Agents for Direct Embodied Manipulation

    Sep 30, 2026Zijie Diao, Yitong Chen, Sicheng Xie +7Long-Horizon Robotic ManipulationContact-Rich Robotic Manipulation