RoboBuilder: Agentic Robot Co-Design with Real-world Video Feedback in the Loop
Organizations: Department of Mechanical Engineering, Massachusetts Institute of Technology Cambridge, MA, USA
Abstract
Can AI agents design robots that work in the real world? We introduce RoboBuilder, a benchmark in which coding agents co-design a robot's hardware and program from standard parts, motors and a custom 3D-printed part, and every design that passes in simulation is built and tested on a physical course. Simulation alone is not enough: under public part data, none of 15 initial passing designs crossed the real course, so an agent learns the simulator from measurements it requests and videos of real runs. Across four frontier coding agents, ten obstacles at three difficulty levels, 16,427 simulations of 12,774 distinct robots and 135 physical robots built in an estimated 240 hours, agents solve nearly all easy obstacles and invent mechanisms our search baselines do not reach, but falter at harder levels, when one robot must cross every obstacle, and in the real world, where 66% of simulated passes hold across all levels. Closing the loop helps: shown a video of its real failure, an agent redesigns the robot, sometimes correcting its simulator, and removes 72% of level-1 and level-2 failures. RoboBuilder measures progress toward agents that engineer real machines. Project Page: https://ghadinehme.com/robobuilder.github.io/
Figures & tables
| Work | Functional task | Real parts | Built for real | New mechanisms | Agent loop | Real feedback |
|---|---|---|---|---|---|---|
| RoboGrammar [ 94 ] | ✓ | ✗ | ✗ | ✗ | ✗ | |
| RoboMorph [ 63 ] | ✓ | ✗ | ✗ | ✓ | ✗ | |
| BesiegeField [ 93 ] | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ |
| Text2Robot [ 68 ] | ✓ | ✓ | ✗ | ✗ | ||
| BrickGPT [ 60 ] | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| Blox-Net [ 22 ] | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ |
Appendix figures & tables48 assets
Supplementary material from the paper’s appendix.
Appendix
| Distance | silhouette | basins/run | never leave | steps switching | new-basin gain | return gain | concept steps |
|---|---|---|---|---|---|---|---|
| Look + parts (main) | 0.64 | 2.0 | 34% | 6.4% | 3.8 (2.1–5.8) | 0.6 | 70% |
| Parts (Jaccard) | 0.66 | 2.0 | 42% | 7.1% | 3.4 (1.9–5.2) | 0.8 | 86% |
| Function | 0.71 | 2.5 | 25% | 11.6% | 3.1 (1.8–4.9) | 1.0 | 76% |
| Agent | Passed | First basin (%) | Basins to pass | Switches that pay (%) |
|---|---|---|---|---|
| Opus 5.5 | 80% | 57 / 63 / 47 | 1.5 / 1.3 / 1.6 | 43 / 35 / 33 |
| Sonnet 5.5 | 64% | 25 / 32 / 29 | 2.2 / 1.9 / 2.3 | 15 / 12 / 23 |
| GPT-6 Astra | 80% | 40 / 43 / 43 | 1.9 / 2.0 / 2.1 | 13 / 20 / 14 |
| GPT-6.1 Sol | 87% | 53 / 50 / 50 | 1.5 / 1.7 / 1.9 | 15 / 23 / 14 |
| All | 78% | 44 / 47 / 42 | 1.7 / 1.7 / 1.9 | 20 / 21 / 20 |
| Agent | steps | same | 10% | 10–50% | concept | plateau P/F | switches P/F |
|---|---|---|---|---|---|---|---|
| Opus 5.5 | 660 | 33% | 38% | 24% | 4.4% | 2 / 39.5 | 0.46 / 0.42 |
| Sonnet 5.5 | 1059 | 53% | 17% | 24% | 5.9% | 3.5 / 42.5 | 0.39 / 0.65 |
| GPT-6 Astra | 808 | 28% | 26% | 36% | 11.0% | 2 / 28 | 1.14 / 1.06 |
| GPT-6.1 Sol | 817 | 31% | 29% | 30% | 9.7% | 3 / 25 | 1.04 / 0.89 |
| All | 3344 | 38% | 26% | 28% | 7.8% | 2 / 33 | 0.83 / 0.74 |
| Agent | wheels | tracks | other | chain | gears | printed | mass (g) | parts | motors |
|---|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | 20 | 2 | 8 | 30% | 100% | 23% | 486 | 135 | 1.9 |
| Sonnet 5.5 | 22 | 1 | 7 | 33% | 93% | 27% | 449 | 73.5 | 1.7 |
| GPT-6 Astra | 21 | 5 | 4 | 40% | 97% | 30% | 523 | 142 | 1.9 |
| GPT-6.1 Sol | 24 | 0 | 6 | 57% | 100% | 30% | 475 | 140 | 1.8 |
| Agent | B mass | C time | tokens in/out (M) | A hours | B+C sims | B+C hours |
|---|---|---|---|---|---|---|
| Opus 5.5 | 0.52 | 0.22 | 16.7/12.4 | 82 | 3000 | 463 |
| Sonnet 5.5 | 0.68 | 0.32 | 11.6/8.5 | 34 | 2689 | 267 |
| GPT-6 Astra | 0.70 | 0.47 | 7.2/3.0 | 58 | 3000 | 326 |
| GPT-6.1 Sol | 0.60 | 0.58 | 12.7/6.6 | 79 | 3000 | 310 |
| Method | Designs | Function NN | Shah | Variety | Concepts | Vendi (function) | Vendi (look) | Non-wheeled |
| Random search | 729 | 1.11 | 0.35 | 10.0 | 12 | 2.2 | 2.0 | 0% |
| CMA-ES | 805 | 0.98 | 0.34 | 7.1 | 9 | 1.9 | 1.8 | 0% |
| Genetic algorithm | 1,136 | 0.94 | 0.38 | 16.6 | 21 | 2.1 | 2.3 | 0% |
| Opus 5.5 | 789 | 19.6 | 0.53 | 50.7 | 135 | 13.0 | 4.0 | 11% |
| Sonnet 5.5 | 454 | 15.4 | 0.49 | 51.1 | 89 | 8.9 | 3.7 | 12% |
| GPT-6 Astra | 511 | 16.9 | 0.52 | 58.6 | 129 | 11.5 | 4.4 | 29% |
| Robots | Pieces | Hours | Print (h) | |
|---|---|---|---|---|
| Simulator learning | 26 | 1,130 | 27 | – |
| Benchmark designs (sim passes) | 87 | 9,390 | 156 | 46 |
| All-in-one robots | 4 | 231 ∗ | 23 | 23 |
| Hardware loop redesigns | 18 | 2,170 | 35 | 37 |
| Total | 135 | 13,000 | 240 | 106 |
| Agent | failures | redesign | vote | real |
|---|---|---|---|---|
| Opus 5.5 | 6 | 5 | 5 | 3/5 |
| Sonnet 5.5 ∗ | 4 | 4 | 4 | 4/4 |
| GPT-6 Astra | 4 | 4 | 4 | 4/4 |
| GPT-6.1 Sol | 5 | 5 | 5 | 3/5 |
| All ∗ | 19 | 18 | 18 | 14/18 |
| Failed partway | Failed early | All | |
|---|---|---|---|
| New concept | 7/7 | 1/2 | 8/9 |
| Adjustment | 1/2 | 5/7 | 6/9 |
| Run | Agent | official | robust | 10/10 |
|---|---|---|---|---|
| Baseline | Opus 5.5 | 5 | 6 | 4 |
| Baseline | Sonnet 5.5 | 3 | 3 | 2 |
| Baseline | GPT-6 Astra | 6 | 7 | 4 |
| Baseline | GPT-6.1 Sol | 5 | 5 | 2 |
| Req.-first | Sonnet 5.5 | 2 | 2 | – |
| Req.-first | GPT-6 Astra | 0 | 7 | 7 |
| Method | crossed /10 | real /10 |
|---|---|---|
| Single agent | 6 | 5 |
| Specialist pools + synthesizer | 7 | – |
| specialists alone (best per obstacle) | 9 | – |
| Cross (%) | Sims | Light | Fast | Score | Novelty | Diversity | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | L1 | L2 | L3 | (%) | (med.) | score | score | (/100) | (function) | (Vendi) |
| Full inventory, random | 0 | 0 | 0 | 0 | – | – | – | 0.0 | 1.1 | 3.4 |
| Full inventory, GA | 0 | 0 | 0 | 0 | – | – | – | 0.0 | 0.9 | 3.2 |
| Parametric, random | 10 | 0 | 0 | 3 | 9 | 0.0 | 0.0 | 0.8 | 1.1 | 2.2 |
| Parametric, CMA-ES | 0 | 0 | 0 | 0 | – | 0.0 | 0.0 | 0.0 | 1.0 | 1.9 |
| Parametric, template | 0 | 0 | 0 | 0 | – | 0.0 | 0.0 | 0.0 | – | – |
| Method | Cross/Print | All | Light | Fast | mean | geo. | rank | 95% CI |
|---|---|---|---|---|---|---|---|---|
| Random search | 0.03 | 0.00 | 0.00 | 0.00 | 0.8 | 1.4 | 7.1 | – |
| CMA-ES | 0.00 | 0.00 | 0.00 | 0.00 | 0.0 | 1.0 | 7.6 | – |
| Template search | 0.00 | 0.00 | 0.00 | 0.00 | 0.0 | 1.0 | 7.6 | – |
| Genetic algorithm | 0.03 | 0.00 | 0.02 | 0.01 | 1.7 | 1.7 | 5.9 | – |
| Single LLM call | 0.07 | 0.00 | 0.00 | 0.00 | 1.7 | 1.6 | 6.8 | – |
| Opus 5.5 | 0.80 | 0.50 | 0.75 | 0.90 | 73.9 | 72.2 | 1.4 | 60.9–87.1 |
| Method | L1 | L2 | L3 | All-obst. |
| Random search (parametric) | 1 | 0 | 0 | – |
| CMA-ES (parametric) | 0 | 0 | 0 | – |
| Template search | 0 | 0 | 0 | – |
| Genetic algorithm | 1 | 0 | 0 | – |
| Single LLM call, Opus 5.5 | 2 | 0 | 0 | 0 |
| Single LLM call, Sonnet 5.5 | 1 | 0 | 0 | 0 |
| Lightest ( Light ) | Fastest ( Fast ) | |||||||
|---|---|---|---|---|---|---|---|---|
| Agent | L1 | L2 | L3 | total | L1 | L2 | L3 | total |
| Opus 5.5 | 9.2 | 8.3 | 5.1 | 22.6 | 9.5 | 9.6 | 8.0 | 27.1 |
| Sonnet 5.5 | 8.4 | 5.3 | 3.5 | 17.2 | 6.4 | 3.1 | 1.5 | 11.0 |
| GPT-6 Astra | 7.4 | 6.5 | 4.2 | 18.1 | 4.1 | 4.0 | 2.0 | 10.1 |
| GPT-6.1 Sol | 7.8 | 4.7 | 6.0 | 18.5 | 4.2 | 2.0 | 3.4 | 9.6 |
| Agent | sim /10 | real /10 |
|---|---|---|
| Opus 5.5 | 5 | 6 |
| Sonnet 5.5 | 3 | 2 |
| GPT-6 Astra | 6 | 5 |
| GPT-6.1 Sol | 5 | 6 |
| L1 | L2 | L3 | ||||
|---|---|---|---|---|---|---|
| Agent | cross | tested | cross | tested | cross | tested |
| Opus 5.5 | 8 | 10 | 4 | 8 | 3 | 6 |
| Sonnet 5.5 | 7 | 9 | 3 | 4 | 1 | 3 |
| GPT-6 Astra | 8 | 10 | 7 | 9 | 2 | 4 |
| GPT-6.1 Sol | 7 | 10 | 5 | 7 | 2 | 6 |
| All | 30 | 39 | 19 | 28 | 8 | 19 |
| before | after | |
|---|---|---|
| Sim passes crossing for real | 0/15 | 57/86 (66%) |
| First-round agent designs predicted | 0/15 | 20/22 |
| Labelled runs predicted | – | 35/38 |
| of which validation | – | 9/10 |
| Measurement | Result |
|---|---|
| Stud pull-off: motor on 4 / 2 studs, hanging 2 4 bricks | 515 g (1.26 N per stud) / about 214 g (1.05 N per stud) |
| Stud peel: long brick on 2 studs, hanging 1 16 beams | 35–47 g at 105 mm (about 36–48 N mm) |
| Reference car tilt test on the course board | slides at 23 ∘ and 15 ∘ (two orientations) |
| Plain brick tilt test on the course board | slides at 23 ∘ ; |
| Friction-pin torque: beams hung on a pinned joint | holds about 4.5 N mm (24.5 assumed before) |
| Lowest speed level that starts a car | level 1 on the flat; level 3 (not 2) on 32 ∘ |
| Physics added | Real observation it explains |
|---|---|
| Stud joints release under overload and on impact | the battery box comes off on a steep ramp |
| Joints peel progressively | the battery stack peels off its stud joints on the steps |
| Loosely built bevel-on-spur meshes slip | the motor whines and the wheels never turn |
| Battery and receiver supply sag under load | fitted to the measured loaded speeds |
| Chains skip teeth above a load per tooth | set from a public filmed lifting test; slack rule |
| Free axles rub under radial load | cars teeter onto free middle wheels over the gap |
| Family | types | pieces |
|---|---|---|
| Beam bricks, liftarms and frames | 29 | 372 |
| Bricks and plates | 20 | 952 |
| Connectors | 13 | 121 |
| Axles | 15 | 263 |
| Pins | 8 | 640 |
| Bushes and CV ball | 3 | 80 |