Can an AI agent turn high-level user intent into a functioning robot and teach it how to behave? We study this question from two complementary aspects: embodiment and behavior creation. For embodiment creation, we investigate how well current multimodal models capture mechanical requirements, and introduce computational metrics for evaluating generated designs before fabrication. For behavior creation, we present Roboforge, an agentic framework that turns task specifications into executable policies through motion generation, policy learning, and execution-grounded revision. Given a robot embodiment and task, the agent generates a reference motion, configures the learning objective, and trains a policy. Crucially, motion remains in the loop: rollout feedback can revise the reference motion, reward, or both. We evaluate a generated embodiment across multiple MLLMs and behavior creation across four diverse robots against reward-refinement and one-shot motion baselines. Our computational metrics expose recurring mechanical failures in generated designs, while behavior experiments show that execution-grounded refinement improves task-level behavior and execution quality. Finally, we fabricate an agent-generated Gorilla and directly deploy learned behaviors on the physical robot.
Figures & tables
Check category
GPT6-Astra
GPT5.6-Sol
GPT5.6-Terra
GPT 5.5
Design and Assembly
6/10
4/10
4/10
3/10
Mechanical and Actuation Feasibility
4/6
0/6
1/6
0/6
Motion and Task Execution
5/7
1/7
1/7
1/7
Table 1: Pre-fabrication feasibility of robot embodiments generated by GPT variants. Each entry x/y denotes the number of passed checks among the y checks in that category. Appendix I.2 provides the complete check definitions and per-design verdicts.
Figure 1: Generated robots (top) and zoomed details (bottom). GPT6-Astra shows the correct mounting-hole layout. The other views expose intersections between the forearm and rear leg in GPT5.6 Sol, the torso and upper arm in GPT 5.5, and the sideplate and elbow motor in GPT5.6 Terra. Surrounding parts are hidden or made transparent to reveal the interfaces.
Figure 2: Overview of RoboForge: motion-guided, execution-grounded policy learning. Given a robot embodiment and task, the agent generates and validates a reference motion, configures the learning objective, and trains a policy. Execution feedback then refines the motion, reward, or both while keeping the embodiment and task fixed.
Method
Go2
MicroDuck
Ostrich
Gorilla
Overall
Eureka-style
2.79
2.01
2.00
1.59
2.10
One-shot Motion
3.93
3.30
4.04
3.14
3.60
RoboForge (Ours)
4.10
4.03
3.99
3.77
3.97
Table 2: User-study ratings across four robot embodiments ( n=57 ; ↑ ). RoboForge achieves the highest overall rating and the highest rating on three of four embodiments.
Method
Yaw Err. (rad)
Foot-slip (m/s)
Action Var. (rad 2 )
Joint Acc. (rad/s 2 )
Eureka-style
0.0345
0.121
0.0830
114
One-shot Motion
0.0584
0.0716
0.00665
31.7
RoboForge (Ours)
0.00836
0.0131
0.00290
22.0
Table 3: Reference-independent execution-quality metrics ( ↓ ). RoboForge achieves the lowest mean value for every metric.
Figure 3: Qualitative comparison of learned policy executions on the gorilla handstand task. Eureka-style fails after an initial rear-leg lift, while One-shot loses balance with outward leg rotation. RoboForge folds the legs inward before inversion, enabling a balanced transition and successful completion.
Agent Behavior
Final Policy Quality
Method
Motion Gen.
Motion Refine
User Rating ↑
Yaw Err. ↓
Foot-slip ↓
Action Var. ↓
Joint Acc. ↓
GPT
16/19
7/19
3.61
0.0172
0.0419
0.0179
29.2
RoboForge (Ours)
19/19 ∗
11/19
3.97
0.00836
0.0131
0.00290
22.0
Table 4: Comparison with unguided GPT. GPT spontaneously introduces and refines reference motions, while our structured pipeline achieves higher user ratings and better execution metrics. ∗ Motion generation is prescribed in our pipeline.
Figure 4: Motion refinement preserves task semantics while improving gait realization. The initial gait has two airborne feet (a), whereas the refined gait uses single-foot swing (b) with coordinated body shift, yielding a more stable four-beat gait.
Figure 5: Refined motions from GPT and RoboForge. Both increase swing-foot clearance, but differ in whole-body coordination. CoM trajectories show that GPT remains nearly straight, whereas RoboForge shifts laterally toward the supporting legs for coordinated weight transfer.
Figure 6: Physical realization of the generated Gorilla and its learned behaviors. Top: hand waving. Bottom: walking. Frames are ordered from left to right.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: GPT-generated robot embodiments used in our experiments. (a) Ostrich. (b) Gorilla, whose design is additionally validated through physical fabrication and assembly.
Figure 8: Fabricated Gorilla embodiment.
Robot
Physics step (s)
Decimation
Policy (Hz)
Go2, MicroDuck, Ostrich
0.005
4
50
Gorilla
0.002
10
50
Appendix
Table 5: Simulation timing shared across methods. All robots are controlled at a policy frequency of 50 Hz.
Parameter
Default value
Parallel environments
4,096
Policy steps per environment per iteration
24
Transitions per iteration
98,304
Optimization epochs per iteration
5
Minibatches per epoch
4
Transitions per minibatch
24,576
Appendix
Table 6: Default PPO and rollout configuration of the shared RL template.
Robot
Walking tasks
Whole-body skills
Total
Go2
High-step walk; low walk
Bow; look around; kick
5
MicroDuck
Fast walk; waddling walk
Bow; look around; kick
5
Ostrich
Happy walk; sad walk
Nod; stretch; squat and rise
5
Gorilla
Alternating walk
Handstand; stretch; wave
4
Total
7 walking tasks
12 skills
19
Appendix
Table 7: Task inventory across four embodiments: seven walking tasks and twelve whole-body skills.
Figure 9: Generated stylized locomotion motions for Ostrich. Left: happy walking ; right: sad walking . Both motions are generated by our method and share the same forward-locomotion objective, while differing in head–neck posture and its coordination with the stepping pattern.
Figure 10: Generated motion for MicroDuck’s kick task produced by our method. The sequence coordinates gaze, lateral body shift, the kicking leg, and the supporting leg across preparation, single-foot support, and recovery.
Score
Description
1
The task is largely unfulfilled, or the motion has severe problems.
2
Few task requirements are fulfilled, or motion problems are substantial.
3
The task is partially or mostly fulfilled, with noticeable shortcomings.
4
The task is fulfilled with generally stable motion and minor problems.
5
The task is fulfilled with smooth, coordinated, and stable motion.
Appendix
Table 8: Five-point rating rubric used in the user study.
Robot
Method
nW
Walking
nS
Non-walking
Overall
Go2
Eureka-style
29
3.00±1.54
43
2.65±1.38
2.79±1.44
One-shot
29
3.00±1.44
43
4.56±0.63
3.93±1.28
RoboForge (Ours)
29
3.76±0.91
43
4.33±0.78
4.10±0.87
MicroDuck
Eureka-style
28
2.64±1.34
43
1.60±0.85
2.01±1.18
One-shot
28
2.82±1.09
43
3.60±1.29
3.30±1.27
RoboForge (Ours)
28
4.00±1.02
43
4.05±0.92
4.03±0.96
Appendix
Table 9: User ratings by robot and task group ( ↑ ), reported as mean ± sample standard deviation. nW and nS denote the numbers of valid individual ratings per method for walking and non-walking tasks, respectively, and are not counts of distinct participants. Bold indicates the highest unrounded mean among the displayed methods within each robot and column, including ties.
Robot
Method
Yaw Err.
Action Var.
Joint Acc.
Foot-slip
(rad)
(rad 2 )
(rad/s 2 )
(m/s)
Go2
Eureka-style
0.00807
0.0531
72.7
0.148
One-shot
0.063
0.00307
23.3
0.2
RoboForge (Ours)
0.0135
0.000174
13.9
0.0354
MicroDuck
Eureka-style
0.0779
0.034
122
0.0564
One-shot
0.139
0.0222
77.9
0.0674
Appendix
Table 10: Physical execution metrics by robot ( ↓ ). Yaw error and slip are averaged over walking tasks, whereas action variation and joint acceleration are averaged over all tasks. Values are rounded to three significant digits. Bold indicates the minimum among the displayed methods within each robot and metric, including ties. Action var. denotes second-order action variation normalized by the number of named joints.
Robot
Method
Walking
Non-walking
Go2
GPT
3.14±1.36
3.86±1.15
RoboForge (Ours)
3.76±0.91
4.33±0.78
MicroDuck
GPT
3.61±1.13
3.91±1.06
RoboForge (Ours)
4.00±1.02
4.05±0.92
Ostrich
GPT
3.21±1.10
4.05±0.83
RoboForge (Ours)
3.57±0.96
4.25±0.84
Appendix
Table 11: User ratings on walking and non-walking tasks ( ↑ ), reported as mean ± sample standard deviation. Bold indicates the higher mean between GPT and RoboForge within each robot and task group.
Robot
Method
Yaw Err.
Action Var.
Joint Acc.
Foot-slip
(rad)
(rad 2 )
(rad/s 2 )
(m/s)
Go2
GPT
0.00746
0.0698
68.9
0.0905
RoboForge (Ours)
0.0135
0.000174
13.9
0.0354
MicroDuck
GPT
0.0133
0.00106
22.3
0.0241
RoboForge (Ours)
0.00435
0.0102
46.4
0.0104
Ostrich
GPT
0.0143
0.00054
17.7
0.0355
Appendix
Table 12: Physical execution metrics of GPT and RoboForge across robot embodiments ( ↓ ). Yaw error and slip are averaged over walking tasks, whereas action variation and joint acceleration are averaged over all tasks. Bold indicates the lower value.
Figure 11: Final learned policy executions for the Go2 low-walk task. Figure 5 shows the corresponding reference-motion revisions, whereas this figure shows the final policies learned from those refined motions. RoboForge more clearly realizes the requested crouched gait, with distinct stepping, sustained low posture, and coordinated support transfer across the gait cycle.
ID
Check
GPT6 Astra
GPT5.6 Sol
GPT5.6 Terra
GPT 5.5
Design and assembly
DA01
Solid validity
Pass
Pass
Pass
Fail
DA02
Manufacturing mesh
Pass
Fail
Fail
Fail
DA03
Export volume fidelity
Pass
Pass
Pass
Pass
DA04
BOM completeness
Pass
Pass
Pass
Pass
DA05
Interface geometry
Pass
Fail
Fail
Fail
Appendix
Table 13: Pass/Fail results for all robot-generation checks. The category counts in the main paper can be recomputed directly from this table.
Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.
Yuyao Liu, Jiayuan Mao, David Hsu +2
Massachusetts Institute of Technology · National University of Singapore · University of Pennsylvania +1
Agentic AI based on Large Language Model generalization capabilities offers a wide range of potential applications, including planning for embodied tasks. For example, embodied agents based on Foundation models can generate plausible plans in autonomous robotics scenarios. Due to limited context windows or hallucinatory phenomena in the next-token prediction formulation, behaviors may be generated without establishing whether the deployed robot and the observed environment actually support the requested operation, in what we call a "grounding failure". Thanks to the recent improvements in reasoning capabilities of foundation models, autonomous robot behavior generation problem can be formulated as a code generation problem. We present iAm.md, a Markdown standard and generation framework, that allows anchoring this process in complementary forms of deployment evidence. Through open-vocabulary semantic mapping, we combine local vision-language detections and object segmentation and refer them to persistent object records in this intermediate standardized representation, allowing agentic introspection. We then study this new technique on a simulated TIAGo, on navigation-and-manipulation tasks, showing how this standardized representation jointly supports skill self-assessment and executable task generalization.
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. AGP achieves success rates of at least 80% in seven of eight task configurations and significantly outperforms previous agentic robot systems. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.
Mengzhao Jia, Yang Lin, Xixin Zhang +3
University of Notre Dame · University of California San Diego · San Diego State University