Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools

Shijia Ge; Alex Zhou; Jianshu Zeng; Yexing Wan; Di Wu; Zelin Zheng; Yazhe Wang; Zhiqi Jia

cs.ROSep 30, 2026

Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools

Authors: Shijia Ge, Alex Zhou, Jianshu Zeng, Yexing Wan, Di Wu, Zelin Zheng, Yazhe Wang, Zhiqi Jia, +7 more

Organizations: Hexafuture Inc. · Peking University · Beijing Institute of Technology · Tsinghua University

Abstract

Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal Robot-Agent Interface), which couples a programming agent that constructs robot tools with an execution agent that uses them in a feedback loop. The programming agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and agents. Across five RoboDojo tasks and four frozen execution agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two agents deciding after each call. Three of the four agents also finish episodes 1.3-1.5 times faster with 1.5-1.7 times fewer execution-agent output tokens; DeepSeek-V4-Flash's cost barely changes. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier agents, organizing robot control around reusable tools that agents can both invoke and revise.

Figures & tables

Figure 1 : URAI overview. (a) A programming agent writes and validates reusable and task-specific tools. An execution agent selects tools from observations and feedback, while a shared GUI/API backend grounds, checks, and executes multi-phase motions locally. Fading outlines in the demonstration-video frames indicate earlier arm poses within one tool call. (b) Decision boundaries: direct control queries the model for each end-effector command; Code as Policies delegates subsequent decisions to generated code, which may query perception; URAI returns control to the agent between tool executions.

Figure 2 : Tool development and execution in URAI. (a) Between episodes, the programming agent revises the tool collection, code, and execution strategy using development feedback. Validated versions are frozen before evaluation; model weights remain fixed. (b) Within an episode, the execution agent selects and parameterizes tools from observations and feedback, while the backend executes each motion locally.

Method	Pour liquid into cup	Pour balls into vase	Fold clothes	Swap blocks a	Stack blocks	Avg. SR $\uparrow$	Time (min) $\downarrow$	Tokens (k) $\downarrow$
$\pi_{0.5}$	28.0	13.3	38.7	0.0	13.3	18.7	–	–
GPT-6 Astra, RoboProbe	8.0	4.0	72.0	0.0	88.0	34.4	–	–
Xiaomi-Robotics-1	64.0	46.0	73.3	0.0	21.3	40.9	–	–
Liber-0 Preview	57.0	31.0	79.0	1.0	72.0	48.0	–	–
Simate-beta	57.3	45.3	80.0	38.7	69.3	58.1	–	–
DeepSeek-V4-Flash, native	0.0	0.0	0.0	0.0	40.0	8.0	21.4	44.3

Table 1 : Success and execution cost on five RoboDojo tasks. Upper block: success rates as posted on the public leaderboard [ 36 ] (official protocol, not paired with ours, no cost data) for the best learned policy of each of the three teams with the highest five-task mean,

\pi_{0.5}

, and GPT-6 Astra through the RoboProbe harness [ 47 ] . Lower block: five episodes per task (seeds 0–4); success is the official terminal state. Avg. SR: average success rate, the mean of the five per-task rates (tasks weighted equally). Time and Tokens average all episodes, failures included, and count execution-agent output with reasoning. Program: one program over the same tools, written in advance and run without the model (time includes both phases). Pour balls uses its task-specific pouring tool. Bold: best success per task over all rows (ties included); within the leaderboard block and within each agent (separated by rules), the best Avg. SR and the lowest time and tokens.

Condition	Motion calls $\downarrow$	s/call	Motion (%)	Obs./probe (%)	Other (%) $\downarrow$	Other (s) $\downarrow$
Claude Fable 5.1 + URAI	4.3	63	31	31	38	426
Claude Fable 5.1, native	22.8	20	34	16	51	772
Claude Opus 5.5 + URAI	3.5	72	37	35	28	250
Claude Opus 5.5, native	21.8	20	44	18	38	405
GPT-6 Astra + URAI	5.6	44	47	21	33	188
GPT-6 Astra, native	21.0	19	50	16	34	276

Table 2 : Execution-time decomposition on RoboDojo. Motion and observation/probe columns give mean per-episode shares of wall time; Other is time outside logged commands. All evaluated episodes, including failures, are included. Bold: within each agent, the fewer motion calls and the less time outside logged commands; the other columns describe the composition of an episode.

Task	Method	Success $\uparrow$	Decisions / model calls $\downarrow$	Tokens $\downarrow$	Time (min) $\downarrow$
Unscrew bottle cap	GPT-Policy, published (R3)	3/3	54.7	–	17.9
Unscrew bottle cap	GPT-6 Astra + URAI	3/3	24.7	1.52k	5.1
Tic-tac-toe	GPT-Policy, published (R8)	3/3	69.7	–	13.6
Tic-tac-toe	GPT-6 Astra + URAI	3/3	37.0	3.26k	5.0
Block into bowl	Robocurve, published	19/20	$\leq$ 20 (budget)	2.1k	2.5
Block into bowl	GPT-6 Astra + URAI	3/3	6.3	443	1.0

Table 3: Real-robot results. Upper block: tasks with a published GPT-6 Astra reference; URAI reports three trials per task with mean time, output tokens, and model calls per episode; the references run on other hardware. Lower block: tasks without a reference; progress is the fraction completed (hot-dog: items of four; toss blocks: blocks of six; fold clothes: operator score); success requires the whole task. Bold: the better value within each task where both rows report a comparable measure (success only at equal trial counts; a call budget is not a measured count).

Figure 3 : Key frames of the seven AgileX tasks under URAI with GPT-6 Astra, initial scene (t1) to completion (t6); chips name the tool called. Fold clothes is the demonstration-video run; other rows are evaluated trials.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

System	Who writes the code, and when	Unit of one model decision	Model at run time	Persistence across episodes	Human interface
Code as Policies [ 26 ]	LM, one program per instruction	the whole program	program decides; perception calls inside	none	–
CaP-X / CaP-Agent0 [ 11 ]	coding agent, one program per turn	one program per turn	re-plans between turns from execution feedback	synthesized skills (CaP-Agent0)	–
Agent as Policy [ 21 ]	preparation agent per task type; execution agent per trial	program and motion commands	revises from physical feedback	saved procedures reused	–
ASPIRE [ 30 ]	agent writes and repairs programs between rollouts	program	program runs	validated skill library	–
ENPIRE [ 42 ]	coding agent revises the policy between rollouts	policy	policy runs	retained policies	harness
LATM / CRAFT [ 5 , 45 ]	tool-maker LLM, offline	one tool call by the tool user	tool user decides each call	toolset	–

Appendix

Table 4 : Closest related systems on the dimensions along which URAI differs, as described in the respective reports.

Figure 4 : Initial scenes of the AgileX dual-arm tasks, seen from the front scene camera and cropped to a common scale: unscrew bottle cap, hot-dog serving, toss blocks into bowl, fold clothes, and pour blocks. Tic-tac-toe and block into bowl are not pictured. Backgrounds are blurred for clarity; the table, arms, and objects are unmodified.

Explore similar work

Jun 15, 2026cs.RO

RHO: Your Coding Agent is Secretly a Roboticist

Code-as-Policies (CaP) has shown that large language models (LLMs) can write code to solve robotics tasks by composing perception, planning, and control primitives. Recent CaP systems, however, rely on multi-turn code-generation loops at test time, which is often infeasible for real-time robot control. We introduce Robotics Harness Optimization (RHO), a novel paradigm in which tool-enabled coding agents, at training time, propose and search for interpretable, neurosymbolic multi-file policy repositories (Repositories-as-Policies) that compose these primitives rather than a single prompt, function, or file. RHO searches with reflective feedback from environment reward and execution rather than teleoperation demonstrations. It generalizes to perturbed pick-and-place settings like LIBERO-PRO, where OpenVLA scores 0.0% and

π_{0.5}

averages 12.83%. Using the same low-level primitives, RHO reaches a 45.0% success rate, 2.5x higher than the strongest multi-turn agentic system, and 3.5x higher than

π_{0.5}

. On Robosuite, RHO sets a new state-of-the-art of 70.0%, exceeding the prior multi-turn record of 68.29% using single-turn execution with no corrective LLM code edits at deployment. When an LLM is used in the control loop, as on RAI's O3DE benchmark, RHO optimizes the deployed agent's multi-file harness of prompts, tools, and control code, improving held-out success from 23.5% to 44.3% with 20% less wall-clock time and 27% fewer tool calls.

Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade +3

University of California, Berkeley · Embodied Science · AMD

Coding Agents Production Agentic Systems Code Generation Neuro-Symbolic Framework

Ask PDF

Oct 1, 2026cs.CV

Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens

Vision language model (VLM) agents can control robots through visual feedback and action primitives, but repeated model invocations and redundant observations incur substantial token overhead. We introduce PyRUA-Lean, an interactive code-execution framework that couples feedback-driven primitive composition with selective observation: the agent composes classical robot primitives and learned vision-language-action (VLA) policies into Python cells that perform conditional checks and local retries, returning only explicitly requested images and state feedback for replanning. Across 700 simulated task instances from LIBERO-PRO, RoboTwin 2.0, and RoboCasa365, we compare PyRUA-Lean with a tool-calling baseline using the same GPT-6 Astra planner and underlying robot primitives. Under equal LLM-call budgets, PyRUA-Lean increases overall success from 63.1% to 71.7%. On instances solved by both agents, it uses 49% fewer LLM calls and 65% fewer input tokens.

Ruiyang Si, Jianxin Bi, Shunyu Yang +9

Peking University · National University of Singapore · NVIDIA +1

Icr-Rl Task Success Rate Robotwin

Ask PDF

Sep 24, 2026cs.RO

RAPID: Robot Agentic Programming from Demonstrations

Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.

Yuyao Liu, Jiayuan Mao, David Hsu +2

Massachusetts Institute of Technology · National University of Singapore · University of Pennsylvania +1

Robot Systems Real-World Manipulation Tasks Action Expert Object-Centric Representations

Ask PDF