Recently, LLM-based autonomous agents have shown strong capabilities in iteratively generating, evaluating, and refining executable solutions. In robotic manipulation, Code-as-Policy lets these agents search over modular and editable robot programs. However, effective program search requires physical understanding: agents must understand how objects can be manipulated, how scene geometry constrains actions, and why interactions succeed or fail. Existing methods rely mainly on execution feedback for program refinement, often leaving these physical questions unresolved. We introduce RoboAuto, a framework for robot program search that explicitly organizes physical understanding at the object, scene, and interaction levels through three components. First, a hierarchical knowledge base combines executable skills with object-centric knowledge of affordances, spatial relations, and action effects, learned from embodied data and online task solving. Second, scene grounding connects this knowledge to the current environment by identifying relevant objects and resolving action-critical quantities. Third, multi-stage planning with self-designed experiments checks whether each stage produces its expected physical effects and actively tests competing hypotheses to resolve physical uncertainties. RoboAuto achieves superior performance across five benchmark families, attaining an 89.1% success rate on short-horizon LIBERO-PRO tasks and more than twice the mean success rate of ASPIRE on LIBERO-PRO Long and LIBERO-Plus. It also outperforms ASPIRE and π0.5 on three challenging long-horizon real-robot tasks requiring precise manipulation. We further show that the knowledge base transfers across simulators and embodiments, and that program search performance scales with its size. Project page: https://roboauto-ai.github.io/
Figures & tables
Figure 2: Overview of RoboAuto. The Hierarchical Knowledge Base (left) stores object-centric physical knowledge, comprising affordances, spatial relations, and action effects, together with executable skills. Physical Scene Grounding (middle) binds the entries relevant to a task to quantities measured in the current scene. Search and Exploration (right) plans the task as stages that end in verified state changes and designs experiments that separate competing physical hypotheses. Interaction evidence updates the knowledge, and verified programs return to the knowledge base as new skills. RoboAuto is evaluated across five benchmark families, LIBERO-PRO, LIBERO-PRO Long, LIBERO-Plus, Robosuite, and RoboTwin 2.0, where RoboTwin 2.0 tests transfer to a new simulator and embodiment.
Figure 3: Unresolved physical assumptions in our baseline runs. In the illustrated ASPIRE and CaP-Agent0 runs, (a) an unsuitable contact strategy persists, (b) a grasp-height rule is transferred without grounding its geometric applicability, and (c) pan placement proceeds without verifying stove activation. The panels show the attempts, outcomes, and retained program choices; Appendix D provides the detailed case studies.
Figure 4: Program success across benchmarks. (a–c) Each dot is a LIBERO-PRO task under position or task perturbation. (d–f) Horizontal bars give each method’s success on every LIBERO-PRO Long, LIBERO-Plus, and Robosuite setting, ordered from the top by RoboAuto’s success, which is printed beside its bar. Dashed lines and their labels give method means.
Figure 5: Ablations of the three components. Mean final-program success over five tasks as Search and Exploration, the Hierarchical Knowledge Base, and Physical Scene Grounding are cumulatively added to the base loop.
Figure 6: Hierarchical knowledge base transfer to RoboTwin 2.0. Success on four RoboTwin 2.0 tasks without retained knowledge and with the hierarchical knowledge base learned from LIBERO. Dashed lines indicate mean success.
Figure 7: Knowledge availability and program search. Mean test success versus search rollouts on the five ablation tasks with 0%, 50%, and 100% of the knowledge base. The 0% and 100% conditions are “+ Search & Exploration” and RoboAuto in Fig. 5 .
Figure 8: Real-robot rollouts. RoboAuto programs on a WidowX AI arm for three tasks. Each row shows four chronological scene-camera keyframes.
Task
RoboAuto
ASPIRE
π0.5
Sort Balls
100.00%
30.00%
0.00%
Stack Dishes
100.00%
3.75%
0.00%
Erase Board
95.83%
0.00%
0.00%
Table 1: Real-robot evaluation. Mean stage-completion score over 20 trials per task. Figure 8 shows example rollouts.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Suite
Settings
Training seeds
Test seeds
Rollouts
LIBERO-PRO Object
20
110000–110014
1–50
45
LIBERO-PRO Goal
20
110000–110014
1–50
45
LIBERO-PRO Spatial
20
110000–110014
1–50
45
LIBERO-PRO Long
10
110000–110014
1–50
55
LIBERO-Plus Spatial
2
110000–110014
1–50
45
LIBERO-Plus Goal
2
110000–110014
1–50
45
Appendix
Table 2: Scene seeds and rollout budgets. Seed intervals include both endpoints. The number of settings counts task–perturbation combinations.
Benchmark
#Set.
RoboAuto
ASPIRE
CaP- Agent0
RHO
π0.5
Same scene layout, same objects
LIBERO-PRO Long
10
73.4
30.8
8.6
–
0.4
LIBERO-Plus
2
80.0
20.0
1.0
–
49.0
All
12
74.5
29.0
7.3
–
8.5
Different scene layout, same objects
LIBERO-PRO Goal
17
77.4
74.0
24.0
50.9
23.1
Appendix
Table 3: Results by similarity to the LIBERO-90 demonstrations. Mean success rate (%) across evaluation settings, grouped by whether their scene layout and manipulated object set match those in the LIBERO-90 demonstrations used for knowledge construction. #Set. denotes the number of evaluation settings in each row.
LIBERO-PRO (60)
LIBERO-PRO Long (10)
LIBERO-Plus (6)
Robosuite
Method
Object
Goal
Spatial
Pos
Task
All
Pos
Task
All
Camera
Layout
All
(7)
π0.5 [ 19 ]
9.1
19.9
10.6
25.2
1.2
13.2
0.3
0.5
0.4
38.0
98.0
68.0
–
CaP-Agent0 [ 13 ]
20.0
21.2
12.9
19.7
16.3
18.0
9.3
7.5
8.6
24.0
42.7
33.3
68.3
RHO [ 11 ]
24.7
53.1
57.3
38.6
51.5
45.0
–
–
–
–
–
–
70.0
ASPIRE [ 25 ]
96.6
63.1
55.3
76.5
66.8
71.7
26.7
37.0
30.8
46.0
43.3
44.7
81.0
RoboAuto (ours)
98.7
73.7
94.8
90.2
87.9
89.1
60.0
93.5
73.4
84.7
96.0
90.3
90.6
Appendix
Table 4: Mean success (%) by benchmark and perturbation. Each mean weights task–perturbation settings equally; setting counts are given under each column group. A dash denotes an unavailable result. The last row gives RoboAuto’s difference from ASPIRE. Per-task results follow in Tables 6 , 7 , 8 , 9 , 10 and 11 .
Suite
RoboAuto
ASPIRE
CaP-Agent0
RHO
π0.5
Object
98.7
96.6
20.0
24.7
9.1
Goal
73.7
63.1
21.2
53.1
19.9
Spatial
94.8
55.3
12.9
57.3
10.6
All settings
89.1
71.7
18.0
45.0
13.2
Appendix
Table 5: LIBERO-PRO suite means. Equal-weight mean success (%) across position and task perturbations.
RoboAuto
ASPIRE
CaP-Agent0
RHO
π0.5
ID
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
1
Place(alphabet_soup, basket)
100
100
100
100
2
4
12
40
0
0
2
Place(bbq_sauce, basket)
92
98
100
100
12
42
32
88
100
2
3
Place(butter, basket)
98
100
100
98
26
18
12
98
54
0
4
Place(chocolate_pudding, basket)
100
96
100
100
18
48
0
0
0
2
5
Place(cream_cheese, basket)
98
98
100
72
12
6
0
0
0
0
Appendix
Table 6: LIBERO-PRO Object. Reported success (%) under position (Pos) and task (Task) perturbations; see Table 2 for RoboAuto’s seed protocol and search budget.
RoboAuto
ASPIRE
CaP-Agent0
RHO
π0.5
ID
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
1
Open(cabinet, drawer_mid)
100
4
72
0
0
0
0
0
0
4
2
Put(bowl, drawer_top)
0
26
86
46
4
0
30
32
94
2
3
Push(plate, stove_front)
0
6
0
76
0
10
0
0
0
0
4
Put(bowl, plate)
100
100
92
88
36
38
98
98
0
2
5
Put(bowl, stove)
98
98
96
0
22
12
100
98
0
4
Appendix
Table 7: LIBERO-PRO Goal. Reported success (%) under position (Pos) and task (Task) perturbations; see Table 2 for RoboAuto’s seed protocol and search budget.
RoboAuto
ASPIRE
CaP-Agent0
RHO
π0.5
ID
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
Pos
Task
1
Pick(between(plate, ramekin), plate)
92
100
94
8
22
14
92
60
2
0
2
Pick(table_center, plate)
98
78
32
64
22
14
96
96
0
2
3
Pick(drawer_top(cabinet_wood), plate)
78
92
10
96
2
10
6
4
0
0
4
Pick(next_to(cookie_box), plate)
100
98
16
94
0
10
0
28
0
2
5
Pick(next_to(plate), plate)
78
98
100
70
10
20
56
64
0
0
Appendix
Table 8: LIBERO-PRO Spatial. Reported success (%) under position (Pos) and task (Task) perturbations; see Table 2 for RoboAuto’s seed protocol and search budget.
Table 12: Component ablation. Reported success (%). KB denotes the hierarchical knowledge base. Each task uses the seeds and budget of its suite in Table 2 .
Figure 9: Per-task ablation trajectories. Test success versus the number of search rollouts as Search and Exploration, the knowledge base (KB), and scene grounding are added cumulatively to the base loop.
Suite
Task
Perturbation
ASPIRE
ASPIRE + KB
RoboAuto
Spatial
Pick(between(plate, ramekin), plate)
Task
8
40
100
Spatial
Pick(next_to(cookie_box), plate)
Pos
16
40
100
Spatial
Pick(next_to(ramekin), plate)
Pos
20
38
98
Spatial
Pick(on(cookie_box), plate)
Pos
24
18
92
Goal
Put(bowl, stove)
Task
0
0
98
Goal
Open(cabinet, drawer_mid)
Task
0
0
4
Appendix
Table 13: Hierarchical knowledge base with ASPIRE. Reported success (%). KB denotes the hierarchical knowledge base. Each task uses the seeds and budget of its suite in Table 2 .
ID
Task
No retained knowledge
Hierarchical knowledge base
1
Stack two bowls
2
88
2
Open microwave
0
96
3
Container on plate
40
92
4
Place empty cup
56
90
Mean
24.5
91.5
Appendix
Table 14: RoboTwin 2.0. Reported success (%) with an 11-rollout search budget and the seed protocol in Table 2 .
Table 15: Knowledge-base size. Reported success (%). KB denotes the hierarchical knowledge base. Each task uses the seeds and budget of its suite in Table 2 .
Figure 10: Per-task knowledge-availability trajectories. Test success versus the number of search rollouts with 0%, randomly sampled 50%, and 100% of the knowledge base entries.
Table 16: Stronger-model comparison. Reported success (%). Each task uses the seeds and budget of its suite in Table 2 .
Task / perturbation
Run 1
Run 2
Run 3
Mean
Sample s.d.
Mug + pudding, task perturbed
94
92
88
91.3
3.1
Pick(next_to(ramekin), plate), camera perturbed
88
92
86
88.7
3.1
Place(chocolate_pudding, basket), task perturbed
96
100
92
96.0
4.0
Appendix
Table 17: Repeated search runs. Each final program is evaluated on test scenes (seeds 1–50). Mean and sample standard deviation are in percentage points.
Figure 11: Token usage and program success on matched tasks. Geometric-mean total tokens per task and mean success on LIBERO-PRO Long and LIBERO-Plus settings with complete token records for RoboAuto, ASPIRE, and CaP-Agent0.
No retained knowledge
Hierarchical knowledge base
ID
Task
Output (M) ↓
Total (M) ↓
Output (M) ↓
Total (M) ↓
1
Stack two bowls
0.190
8.780
0.210
18.620
2
Open microwave
0.280
13.710
0.420
33.320
3
Container on plate
0.180
8.580
0.280
29.890
4
Place empty cup
0.080
6.170
0.120
12.640
Mean
0.183
9.310
0.258
23.618
Appendix
Table 18: RoboTwin 2.0 token usage. Output and total counts are in millions. Task IDs correspond to Table 14 .
Table 20: LIBERO-Plus token usage. Output and total counts are in millions. Task IDs correspond to Table 10 .
Figure 12: Constructing object-centric knowledge from embodied data. A vision-language model converts each demonstration video into an evidence record containing entities, temporally localized events, and an episode outcome. The example shows the LIBERO-90 task “close the microwave,” in which event E06 moves the door from open to flush. Knowledge distilled across such records is organized into affordances, spatial relations, and action effects. The expanded affordance entry cites supporting events from multiple episodes and declares the scene-dependent quantities, shown in teal, that must be measured again when the entry is applied in a new scene.
Figure 13: Search and Exploration. The coder decomposes the task into stages with intended object states and executable verifiers (1), and the environment executes these stages sequentially (2). When an execution fails, an independent auditor evaluates the stage outcomes from synchronized videos and may reject a state that the program verifier accepted (3). The earliest incomplete stage is returned to the coder (4), which revises the stage or replans the task before executing the next candidate program (5). If the cause remains uncertain, the coder branches into a separately implemented experiment that collects targeted physical evidence before revision continues. Programs verified across the search pool contribute reusable executable skills.
Figure 14: The four component variants on turning on the stove and placing the pan (LIBERO-PRO Long, task perturbation of turning on the stove and placing the moka pot). Each panel adds one component to the preceding variant. The base loop rewrites the complete program from a per-scene reward. Search and Exploration plans stages whose verdicts localize the failure. The Hierarchical Knowledge Base supplies knob and handle contacts in the first turn but estimates their quantities from segmentation-mask spread and fixed offsets. Physical Scene Grounding measures the knob height and pan geometry and verifies burner activation before the pan is moved.
Figure 15: Book into the caddy (LIBERO-PRO Long, position perturbation). CaP-Agent0 continues using top-down grasps after its first approach topples the book. RoboAuto selects a cover pinch from the thin-slab entry and grounds the quantities required to execute it.
Figure 16: Placing the red mug on the plate and the pudding to its right (LIBERO-PRO Long, task perturbation of placing the white mug on the plate and the pudding to its right). ASPIRE applies a grasp-height rule stored in a skill learned on a different mug, causing the gripper to close on air in approximately half of the training scenes. RoboAuto measures the rim of the current mug and places the pads on it.
Figure 17: Turning on the stove and placing the pan (LIBERO-PRO Long, task perturbation of turning on the stove and placing the moka pot). CaP-Agent0 rotates the knob by a stored angle and proceeds without checking the burner. RoboAuto’s knob entry specifies the expected observable effect, which the program verifies before beginning pan placement.
Figure 18: Real-robot rollouts in detail. RoboAuto programs on a WidowX AI arm for three tasks. Each row shows nine scene-camera keyframes in time order and one wrist-camera frame. Orange labels mark actions in progress, and checked labels mark stages verified before the next stage begins.