Recently, LLM-based autonomous agents have shown strong capabilities in iteratively generating, evaluating, and refining executable solutions. In robotic manipulation, Code-as-Policy lets these agents search over modular and editable robot programs. However, effective program search requires physical understanding: agents must understand how objects can be manipulated, how scene geometry constrains actions, and why interactions succeed or fail. Existing methods rely mainly on execution feedback for program refinement, often leaving these physical questions unresolved. We introduce RoboAuto, a framework for robot program search that explicitly organizes physical understanding at the object, scene, and interaction levels through three components. First, a hierarchical knowledge base combines executable skills with object-centric knowledge of affordances, spatial relations, and action effects, learned from embodied data and online task solving. Second, scene grounding connects this knowledge to the current environment by identifying relevant objects and resolving action-critical quantities. Third, multi-stage planning with self-designed experiments checks whether each stage produces its expected physical effects and actively tests competing hypotheses to resolve physical uncertainties. RoboAuto achieves superior performance across five benchmark families, attaining an 89.1% success rate on short-horizon LIBERO-PRO tasks and more than twice the mean success rate of ASPIRE on LIBERO-PRO Long and LIBERO-Plus. It also outperforms ASPIRE and π0.5 on three challenging long-horizon real-robot tasks requiring precise manipulation. We further show that the knowledge base transfers across simulators and embodiments, and that program search performance scales with its size. Project page: https://roboauto-ai.github.io/
Figures & tables
Figure 2: Overview of RoboAuto. The Hierarchical Knowledge Base (left) stores object-centric physical knowledge, comprising affordances, spatial relations, and action effects, together with executable skills. Physical Scene Grounding (middle) binds the entries relevant to a task to quantities measured in the current scene. Search and Exploration (right) plans the task as stages that end in verified state changes and designs experiments that separate competing physical hypotheses. Interaction evidence updates the knowledge, and verified programs return to the knowledge base as new skills. RoboAuto is evaluated across five benchmark families, LIBERO-PRO, LIBERO-PRO Long, LIBERO-Plus, Robosuite, and RoboTwin 2.0, where RoboTwin 2.0 tests transfer to a new simulator and embodiment.
Figure 3: Unresolved physical assumptions in our baseline runs. In the illustrated ASPIRE and CaP-Agent0 runs, (a) an unsuitable contact strategy persists, (b) a grasp-height rule is transferred without grounding its geometric applicability, and (c) pan placement proceeds without verifying stove activation. The panels show the attempts, outcomes, and retained program choices; Appendix D provides the detailed case studies.
Figure 4: Program success across benchmarks. (a–c) Each dot is a LIBERO-PRO task under position or task perturbation. (d–f) Horizontal bars give each method’s success on every LIBERO-PRO Long, LIBERO-Plus, and Robosuite setting, ordered from the top by RoboAuto’s success, which is printed beside its bar. Dashed lines and their labels give method means.
Figure 5: Ablations of the three components. Mean final-program success over five tasks as Search and Exploration, the Hierarchical Knowledge Base, and Physical Scene Grounding are cumulatively added to the base loop.
Figure 6: Hierarchical knowledge base transfer to RoboTwin 2.0. Success on four RoboTwin 2.0 tasks without retained knowledge and with the hierarchical knowledge base learned from LIBERO. Dashed lines indicate mean success.
Figure 7: Knowledge availability and program search. Mean test success versus search rollouts on the five ablation tasks with 0%, 50%, and 100% of the knowledge base. The 0% and 100% conditions are “+ Search & Exploration” and RoboAuto in Fig. 5 .
Figure 8: Real-robot rollouts. RoboAuto programs on a WidowX AI arm for three tasks. Each row shows four chronological scene-camera keyframes.
Task
RoboAuto
ASPIRE
π0.5
Sort Balls
100.00%
30.00%
0.00%
Stack Dishes
100.00%
3.75%
0.00%
Erase Board
95.83%
0.00%
0.00%
Table 1: Real-robot evaluation. Mean stage-completion score over 20 trials per task. Figure 8 shows example rollouts.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Suite
Settings
Training seeds
Test seeds
Rollouts
LIBERO-PRO Object
20
110000–110014
1–50
45
LIBERO-PRO Goal
20
110000–110014
1–50
45
LIBERO-PRO Spatial
20
110000–110014
1–50
45
LIBERO-PRO Long
10
110000–110014
1–50
55
LIBERO-Plus Spatial
2
110000–110014
1–50
45
LIBERO-Plus Goal
2
110000–110014
1–50
45
Appendix
Table 2: Scene seeds and rollout budgets. Seed intervals include both endpoints. The number of settings counts task–perturbation combinations.
Benchmark
#Set.
RoboAuto
ASPIRE
CaP- Agent0
RHO
π0.5
Same scene layout, same objects
LIBERO-PRO Long
10
73.4
30.8
8.6
–
0.4
LIBERO-Plus
2
80.0
20.0
1.0
–
49.0
All
12
74.5
29.0
7.3
–
8.5
Different scene layout, same objects
LIBERO-PRO Goal
17
77.4
74.0
24.0
50.9
23.1
Appendix
Table 3: Results by similarity to the LIBERO-90 demonstrations. Mean success rate (%) across evaluation settings, grouped by whether their scene layout and manipulated object set match those in the LIBERO-90 demonstrations used for knowledge construction. #Set. denotes the number of evaluation settings in each row.
LIBERO-PRO (60)
LIBERO-PRO Long (10)
LIBERO-Plus (6)
Robosuite
Method
Object
Goal
Spatial
Pos
Task
All
Pos
Task
All
Camera
Layout
All
(7)
π0.5 [ 19 ]
9.1
19.9
10.6
25.2
1.2
13.2
0.3
0.5
0.4
38.0
98.0
68.0
–
CaP-Agent0 [ 13 ]
20.0
21.2
12.9
19.7
16.3
18.0
9.3
7.5
8.6
24.0
42.7
33.3
68.3
RHO [ 11 ]
24.7
53.1
57.3
38.6
51.5
45.0
–
–
–
–
–
–
70.0
ASPIRE [ 25 ]
96.6
63.1
55.3
76.5
66.8
71.7
26.7
37.0
30.8
46.0
43.3
44.7
81.0
RoboAuto (ours)
98.7
73.7
94.8
90.2
87.9
89.1
60.0
93.5
73.4
84.7
96.0
90.3
90.6
Appendix
Table 4: Mean success (%) by benchmark and perturbation. Each mean weights task–perturbation settings equally; setting counts are given under each column group. A dash denotes an unavailable result. The last row gives RoboAuto’s difference from ASPIRE. Per-task results follow in Tables 6 , 7 , 8 , 9 , 10 and 11 .
Suite
RoboAuto
ASPIRE
CaP-Agent0
RHO
π0.5
Object
98.7
96.6
20.0
24.7
9.1
Goal
73.7
63.1
21.2
53.1
19.9
Spatial
94.8
55.3
12.9
57.3
10.6
All settings
89.1
71.7
18.0
45.0
13.2
Appendix
Table 5: LIBERO-PRO suite means. Equal-weight mean success (%) across position and task perturbations.
RoboAuto
ASPIRE
CaP-Agent0
RHO
π0.5
ID
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
1
Place(alphabet_soup, basket)
100
100
100
100
2
4
12
40
0
0
2
Place(bbq_sauce, basket)
92
98
100
100
12
42
32
88
100
2
3
Place(butter, basket)
98
100
100
98
26
18
12
98
54
0
4
Place(chocolate_pudding, basket)
100
96
100
100
18
48
0
0
0
2
5
Place(cream_cheese, basket)
98
98
100
72
12
6
0
0
0
0
Appendix
Table 6: LIBERO-PRO Object. Reported success (%) under position (Pos) and task (Task) perturbations; see Table 2 for RoboAuto’s seed protocol and search budget.
RoboAuto
ASPIRE
CaP-Agent0
RHO
π0.5
ID
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
1
Open(cabinet, drawer_mid)
100
4
72
0
0
0
0
0
0
4
2
Put(bowl, drawer_top)
0
26
86
46
4
0
30
32
94
2
3
Push(plate, stove_front)
0
6
0
76
0
10
0
0
0
0
4
Put(bowl, plate)
100
100
92
88
36
38
98
98
0
2
5
Put(bowl, stove)
98
98
96
0
22
12
100
98
0
4
Appendix
Table 7: LIBERO-PRO Goal. Reported success (%) under position (Pos) and task (Task) perturbations; see Table 2 for RoboAuto’s seed protocol and search budget.
RoboAuto
ASPIRE
CaP-Agent0
RHO
π0.5
ID
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
1
Pick(between(plate, ramekin), plate)
92
100
94
8
22
14
92
60
2
0
2
Pick(table_center, plate)
98
78
32
64
22
14
96
96
0
2
3
Pick(drawer_top(cabinet_wood), plate)
78
92
10
96
2
10
6
4
0
0
4
Pick(next_to(cookie_box), plate)
100
98
16
94
0
10
0
28
0
2
5
Pick(next_to(plate), plate)
78
98
100
70
10
20
56
64
0
0
Appendix
Table 8: LIBERO-PRO Spatial. Reported success (%) under position (Pos) and task (Task) perturbations; see Table 2 for RoboAuto’s seed protocol and search budget.
Table 12: Component ablation. Reported success (%). KB denotes the hierarchical knowledge base. Each task uses the seeds and budget of its suite in Table 2 .
Figure 9: Per-task ablation trajectories. Test success versus the number of search rollouts as Search and Exploration, the knowledge base (KB), and scene grounding are added cumulatively to the base loop.
Suite
Task
Perturbation
ASPIRE
ASPIRE + KB
RoboAuto
Spatial
Pick(between(plate, ramekin), plate)
Task
8
40
100
Spatial
Pick(next_to(cookie_box), plate)
Pos
16
40
100
Spatial
Pick(next_to(ramekin), plate)
Pos
20
38
98
Spatial
Pick(on(cookie_box), plate)
Pos
24
18
92
Goal
Put(bowl, stove)
Task
0
0
98
Goal
Open(cabinet, drawer_mid)
Task
0
0
4
Appendix
Table 13: Hierarchical knowledge base with ASPIRE. Reported success (%). KB denotes the hierarchical knowledge base. Each task uses the seeds and budget of its suite in Table 2 .
ID
Task
No retained knowledge
Hierarchical knowledge base
1
Stack two bowls
2
88
2
Open microwave
0
96
3
Container on plate
40
92
4
Place empty cup
56
90
Mean
24.5
91.5
Appendix
Table 14: RoboTwin 2.0. Reported success (%) with an 11-rollout search budget and the seed protocol in Table 2 .
Table 15: Knowledge-base size. Reported success (%). KB denotes the hierarchical knowledge base. Each task uses the seeds and budget of its suite in Table 2 .
Figure 10: Per-task knowledge-availability trajectories. Test success versus the number of search rollouts with 0%, randomly sampled 50%, and 100% of the knowledge base entries.
Table 16: Stronger-model comparison. Reported success (%). Each task uses the seeds and budget of its suite in Table 2 .
Task / perturbation
Run 1
Run 2
Run 3
Mean
Sample s.d.
Mug + pudding, task perturbed
94
92
88
91.3
3.1
Pick(next_to(ramekin), plate), camera perturbed
88
92
86
88.7
3.1
Place(chocolate_pudding, basket), task perturbed
96
100
92
96.0
4.0
Appendix
Table 17: Repeated search runs. Each final program is evaluated on test scenes (seeds 1–50). Mean and sample standard deviation are in percentage points.
Figure 11: Token usage and program success on matched tasks. Geometric-mean total tokens per task and mean success on LIBERO-PRO Long and LIBERO-Plus settings with complete token records for RoboAuto, ASPIRE, and CaP-Agent0.
No retained knowledge
Hierarchical knowledge base
ID
Task
Output (M) ↓
Total (M) ↓
Output (M) ↓
Total (M) ↓
1
Stack two bowls
0.190
8.780
0.210
18.620
2
Open microwave
0.280
13.710
0.420
33.320
3
Container on plate
0.180
8.580
0.280
29.890
4
Place empty cup
0.080
6.170
0.120
12.640
Mean
0.183
9.310
0.258
23.618
Appendix
Table 18: RoboTwin 2.0 token usage. Output and total counts are in millions. Task IDs correspond to Table 14 .
Table 20: LIBERO-Plus token usage. Output and total counts are in millions. Task IDs correspond to Table 10 .
Figure 12: Constructing object-centric knowledge from embodied data. A vision-language model converts each demonstration video into an evidence record containing entities, temporally localized events, and an episode outcome. The example shows the LIBERO-90 task “close the microwave,” in which event E06 moves the door from open to flush. Knowledge distilled across such records is organized into affordances, spatial relations, and action effects. The expanded affordance entry cites supporting events from multiple episodes and declares the scene-dependent quantities, shown in teal, that must be measured again when the entry is applied in a new scene.
Figure 13: Search and Exploration. The coder decomposes the task into stages with intended object states and executable verifiers (1), and the environment executes these stages sequentially (2). When an execution fails, an independent auditor evaluates the stage outcomes from synchronized videos and may reject a state that the program verifier accepted (3). The earliest incomplete stage is returned to the coder (4), which revises the stage or replans the task before executing the next candidate program (5). If the cause remains uncertain, the coder branches into a separately implemented experiment that collects targeted physical evidence before revision continues. Programs verified across the search pool contribute reusable executable skills.
Figure 14: The four component variants on turning on the stove and placing the pan (LIBERO-PRO Long, task perturbation of turning on the stove and placing the moka pot). Each panel adds one component to the preceding variant. The base loop rewrites the complete program from a per-scene reward. Search and Exploration plans stages whose verdicts localize the failure. The Hierarchical Knowledge Base supplies knob and handle contacts in the first turn but estimates their quantities from segmentation-mask spread and fixed offsets. Physical Scene Grounding measures the knob height and pan geometry and verifies burner activation before the pan is moved.
Figure 15: Book into the caddy (LIBERO-PRO Long, position perturbation). CaP-Agent0 continues using top-down grasps after its first approach topples the book. RoboAuto selects a cover pinch from the thin-slab entry and grounds the quantities required to execute it.
Figure 16: Placing the red mug on the plate and the pudding to its right (LIBERO-PRO Long, task perturbation of placing the white mug on the plate and the pudding to its right). ASPIRE applies a grasp-height rule stored in a skill learned on a different mug, causing the gripper to close on air in approximately half of the training scenes. RoboAuto measures the rim of the current mug and places the pads on it.
Figure 17: Turning on the stove and placing the pan (LIBERO-PRO Long, task perturbation of turning on the stove and placing the moka pot). CaP-Agent0 rotates the knob by a stored angle and proceeds without checking the burner. RoboAuto’s knob entry specifies the expected observable effect, which the program verifies before beginning pan placement.
Figure 18: Real-robot rollouts in detail. RoboAuto programs on a WidowX AI arm for three tasks. Each row shows nine scene-camera keyframes in time order and one wrist-camera frame. Orange labels mark actions in progress, and checked labels mark stages verified before the next stage begins.
Traditional robot programming is notoriously challenging: it requires orchestrating multimodal perception, managing physical contact dynamics, and handling diverse environment configurations and execution failures. We introduce ASPIRE (Agentic Skill Programming through Iterative Robot Exploration), a continual learning system that autonomously writes and refines robot control programs in a code-as-policy paradigm while compounding experience into a reusable skill library. ASPIRE operates in an open-ended learning loop, consisting of three components: (1) a closed-loop robot execution engine that exposes fine-grained multimodal traces, enabling the agent to autonomously diagnose failures, synthesize repairs, and validate outcomes; (2) a continually expanding skill library that distills validated fixes into reusable, transferable robotic knowledge; and (3) an evolutionary search procedure that generates diverse programs, systematically debugging them to explore beyond single-trajectory refinement. ASPIRE surpasses prior methods by up to 77% on manipulation tasks under perturbation, 72% on Robosuite's bimanual handover task, and up to 32% on long-horizon household tasks. The accumulated skill library further enables zero-shot generalization: on representative unseen long-horizon tasks, ASPIRE achieves 31% success, outperforming the 4% success rate of prior methods. Performance scales with the capabilities of frontier models. With Claude Opus 5.5, ASPIRE nearly saturates these benchmarks, achieving 99% average success across all 80 LIBERO-Pro tasks and 100% success on representative Robosuite and BEHAVIOR-1K tasks. Beyond simulation, skills discovered in simulation reduce real-world robot programming effort across embodiments and robot APIs, while skills acquired through online interaction with physical robots enable ASPIRE to solve a Rubik's Cube with progressively improving efficiency.
Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.
Yuyao Liu, Jiayuan Mao, David Hsu +2
Massachusetts Institute of Technology · National University of Singapore · University of Pennsylvania +1
General-purpose agents can plan, use tools, and revise their behavior from feedback, but it remains unclear whether these capabilities transfer from digital environments to embodied manipulation. To investigate this question, we introduce LIBERO-Agent, an agent-native benchmark for evaluating these agents in robot manipulation tasks. Rather than asking agents to submit task-level Python control programs or operate through high-level robot skills, LIBERO-Agent provides an interactive robotic environment where agents can select which observations to inspect, process them with their own tools, and issue native action commands. LIBERO-Agent integrates 200 tasks into a common interaction framework and provides a 30-task primary suite that separates perception, short-horizon execution, and long-horizon composition. Results reveal a pronounced reliability gap: while agents perform well on perception and easy short-horizon tasks, their performance degrades substantially on hard short-horizon and long-horizon tasks. Richer observations improve short-horizon manipulation, while demonstration benefits depend on the agent and format. Among these agents, GPT-6 Astra achieves the strongest overall performance. Further analysis shows its major advantage lies in mechanism interaction, especially when sustained physical contact is needed, while its remaining failures stem from cross-stage interference and geometric errors.
Zijie Diao, Yitong Chen, Sicheng Xie +7
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute