cs.ROOct 1, 2026
SaveOpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies
Organizations: University College London · Carnegie Mellon University
Abstract
Coding agents are extending their reach into the physical world by writing and executing robot control programs. One might expect the agents to use the existing mature software stack that engineers have developed over decades to access sensors and control motion. Yet prior work primarily engineers complex custom harnesses to orchestrate agents for robot use, particularly by prescribing specialized workflows and providing bespoke interfaces. This raises the question: "Is such additional harness engineering necessary?" We introduce OpenRUA, a zero-abstraction harness that bypasses bespoke abstraction layers by providing off-the-shelf coding agents with only terminal access to the robot's native software interface ROS 2. OpenRUA employs a minimalist workspace-as-harness design, only offering ROS 2 documentation and basic tools while leaving the coding agent to organize its own work without orchestrating any agentic workflow. Within this workspace, OpenRUA recasts perception as file I/O and manipulation as coding. With Claude Code powered by Claude Opus 5, OpenRUA achieves success rates of 99.0% on CaP-Bench and 87.0% on LIBERO-PRO, demonstrating that an off-the-shelf coding agent can serve as a zero-shot visuomotor policy through the robot's native interface, without bespoke primitives or task-specific training. Under this minimalist design, further analysis reveals striking emergent behaviors of coding agents: (1) For perception, the agent spontaneously writes programs that process raw sensory inputs and derive metric measurements in 96.80% of episodes. (2) For manipulation, the agent spontaneously builds motion-control clients (e.g., gripper control) in 95.87% of episodes and closed-loop control programs (e.g., adjusting motion based on sensor feedback) in 50.13% of episodes. Our code is available at https://github.com/terminalworld/OpenRUA.
Figures & tables
Figure 1: OpenRUA: a minimalist workspace-as-harness for robot-use agents. Given a task such as placing the block in the bowl, OpenRUA enables any off-the-shelf coding agent to operate a real or simulated robot through its native ROS 2 interface, bypassing bespoke abstraction layers. The workspace serves as the harness by only providing ROS 2 documentation and basic tools, making robot perception accessible through file I/O and manipulation through coding, without orchestrating any agentic workflow. Within this workspace, the agent exhibits emergent perception and control capabilities as it works to complete the task. It writes and executes ros2cli commands or rclpy programs that send control requests to the robot’s controllers, here grasping and lifting the wooden block. The agent saves new RGB-D observations as workspace files and uses images, computed measurements, and terminal feedback to autonomously verify progress and refine its actions until the block is placed in the bowl.
Figure 2: A minimalist workspace-as-harness for robot use. (a) The opening prompt specifies the task and asks the agent to explore the machine, complete the task, and verify the result. (b) The workspace offers documentation and tools for getting started and stores the observations, programs, and records the agent produces during execution.
Table 1: OpenRUA achieves 99.0% success on CaP-Bench and 87.0% on LIBERO-PRO in a zero-shot setting. Train. , Exp. , and Oracle indicate training on robot demonstrations, experience accumulated before evaluation, and access to the benchmark’s success predicate during execution, respectively. Reasoning effort is shown in parentheses; CC denotes Claude Code. Baseline sources and evaluation protocols are detailed in Sections C.4 and C .
| Benchmark | Episodes | Image processing (%) | Metric programs (%) |
| CaP-Bench | 700 | 47.43% | 97.43% |
| LIBERO-PRO | 800 | 94.75% | 96.25% |
| Overall | 1,500 | 72.67% | 96.80% |
Table 2: The agent writes its own programs to process sensor images and compute metric measurements. Percentage of episodes in which Claude Code with Opus 5 processes sensor images with self-written code or runs a self-written metric program. All successful and failed episodes are included. More analysis is provided in Appendix E .
Figure 3: An agent-written program for locating the nut and pegs from depth . See Appendix I for further code details.
Figure 4: An example of agent-built perception in the CaP-Bench Nut task. The agent combines visual understanding with numerical processing: (a) it reads the overhead camera image to identify the nut and two pegs; (b) it segments the depth array by height and uses camera calibration to compute their positions in world coordinates; (c) it enlarges a crop of the nut and adds a pixel grid to inspect the handle’s orientation and width; and (d) it checks the grasp using the wrist camera image together with the measured finger gap.
Figure 5: The agent writes its own code to reconstruct scene geometry from depth and create views tailored to its own visual understanding. In a RoboCasa365 dishwasher task, (a) it reads the wrist-camera image to identify the rack and its rail. (b) It projects depth points onto the counter plane and colors them by height. (c) It renders the reconstructed point cloud from a chosen viewpoint to inspect the rack’s structure. (d) It slices the point cloud along one axis and plots cross-sections to locate the rail along which the rack moves.
| Benchmark | Episodes | Motion-control clients (%) | Feedback-control programs (%) |
| CaP-Bench | 700 | 97.00% | 60.14% |
| LIBERO-PRO | 800 | 94.88% | 41.38% |
| Overall | 1,500 | 95.87% | 50.13% |
Table 3: The agent writes its own motion-control clients and feedback-control programs. Percentage of episodes in which Claude Code with Opus 5 writes a motion-control client or executes a self-written program that uses sensor feedback to adjust motion or determine when to stop. All successful and failed episodes are included. More analysis is provided in Appendix E .
Figure 6: The agent constructs motion-control clients and feedback logic during a LIBERO-PRO drawer task. (a) It inspects the robot’s interfaces and acquires observations. (b) It writes a reusable client library for inverse kinematics, trajectory and gripper actions, and velocity commands. (c) It writes a pulling program that checks finger gap and displacement, stopping if the grip is lost or motion stalls. (d) It executes its programs and reads the resulting measurements to assess progress. Some lines are omitted to highlight the key operations. See Appendix I for the programs and Section E.6 for execution details.
Figure 7: The agent revises its grasp after a failed attempt in a LIBERO-PRO task. (a) It compares camera frames and detects that the scene has not settled. (b) Its first grasp misses and knocks the ketchup bottle over; finger-gap readings and a camera view reveal the failure. (c) It uses a wrist-camera observation and a reconstructed point cloud to estimate the fallen bottle’s orientation and plan a new grasp. (d) It places the bottle upright in the basket and checks the result. The panels show RGB-D observations processed by agent-written code.
| Method | Setting | Success Rate (%) | ||||||||
| Agent | Model (Effort) | Train. | Exp. | Oracle | Atomic-Seen | Comp.-Seen | Comp.-Unseen | Overall | ||
| Learned Policy | GR00T N1.5 | – | – | ✓ | ✗ | ✗ | 41.9 | 0.0 | 0.2 | 15.1 |
| – | – | ✓ | ✓ | ✗ | 68.5 | 40.6 | 42.1 | 51.1 | ||
| RLDX-1 | – | – | ✓ | ✗ | ✗ | 60.0 | 21.3 | 5.0 | 30.0 | |
| Agent + Policy | Harness VLA | Codex | GPT-5.5 (xhigh) | ✓ | ✓ | ✓ | 92.0 | 61.0 | 13.8 | 57.1 |
| CC | Opus 4.7 (max) | ✓ | ✓ | ✓ | 79.4 | 47.5 | 15.0 | 48.6 | ||
Table 4: OpenRUA generalizes to unseen composite tasks in RoboCasa365. Success rates (%) are reported for atomic tasks and composite tasks whose templates are included in (Comp.-Seen) or held out from (Comp.-Unseen) the benchmark’s pretraining data. Detailed evaluation protocols are provided in Appendix C .
Figure 8: Success improves with test-time computation but plateaus at different levels across models. Success rates on the LIBERO-10 suite of LIBERO-PRO are plotted against agent-turn and output-token budgets, with 200 trials per configuration across both perturbation cells. A trial counts as solved within a budget if the benchmark predicate first held before it; one turn is one model response, and tokens are those the model wrote, including thinking. The right end of each curve is the success rate reported in Table 11 ; the runs behind the curves, and how the token position of a Claude Code episode is estimated, are described in Appendix F .
| Cause of failure | LIBERO-PRO | CaP-Bench | RoboCasa365 | All |
| A Success criterion not stated in the instruction | 42 | 1 | 36 | 79 |
| B Predicate contradicts the instruction | 17 | 0 | 2 | 19 |
| C Budget exhausted while handling undocumented machine behavior | 15 | 0 | 125 | 140 |
| D Object or force beyond the machine’s limits | 12 | 0 | 63 | 75 |
| E Perception, control, or waiting error of the agent | 18 | 6 | 76 | 100 |
| Analyzed failed episodes | 104 | 7 | 302 | 413 |
Table 5: Task criteria and undocumented machine behavior account for over half of the analyzed failures. Counts of failed episodes with Claude Code and Opus 5 across CaP-Bench, LIBERO-PRO, and RoboCasa365, assigned one category each by transcript and workspace analysis. Classification procedures, limitations, and sub-causes are provided in Appendices G and 12 .
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Images read | Processed images | Metric programs | Targets traced to computed coordinates |
| CaP-Bench | 14 | 16% | 13 | 94% |
| LIBERO-PRO | 21 | 36% | 21 | 93% |
| RoboCasa365 | 31 | 40% | 35 | 90% |
Table 6: Most Cartesian motion targets trace to programmatically computed coordinates. Statistics cover all mainline episodes with Claude Code and Opus 5 across the three benchmarks. Images read and Metric programs are median counts per episode: image-file Read calls and executions of agent-written geometric measurement programs, respectively. Processed images is the share of image reads attributed to views processed from sensor data by agent-written code, such as crops, zooms, grids, and overlays. Targets traced is the share of Cartesian-target motion commands whose coordinates match an earlier metric-program output within 1.5 mm or are computed inside the motion program itself.
| Purpose | Processed images | |||||||
| Benchmark | Scene | Verify | Inspect | Crop / zoom | Grid / markers | Depth view | Mosaic | Mask overlay |
| CaP-Bench | 32% | 37% | 31% | 80% | 3% | 3% | 1% | 2% |
| LIBERO-PRO | 38% | 35% | 27% | 94% | 11% | 4% | 2% | 2% |
| RoboCasa365 | 18% | 47% | 36% | 68% | 10% | 5% | 6% | 2% |
Table 7: Crops and enlarged views dominate the processed images the agent reads. For Claude Code with Opus 5 across all mainline episodes, read purposes are reported as shares of all image reads and are inferred from preceding events. Processing categories are shares of reads of processed images and can overlap; a crop with a grid counts in both categories.
| Technique | CaP-Bench | LIBERO-PRO | RoboCasa365 |
| Depth threshold / height band | 97% | 99% | 98% |
| Centroid / bounding box | 100% | 96% | 94% |
| Point cloud / plane fit | 83% | 69% | 88% |
| Connected components / contours | 68% | 69% | 56% |
| Color threshold | 48% | 24% | 41% |
| Crop / zoom for its own reading | 60% | 99% | 98% |
Table 8: Agent-written perception code uses a range of geometric and image-processing techniques. Each percentage is the share of mainline episodes with Claude Code and Opus 5 in which the technique appears in agent-written code. Techniques can co-occur; code presence does not establish successful execution.
Figure 9: The agent writes image-processing code to check whether a switch moved. In a RoboCasa365 electric-kettle task, (a) it locates the kettle and switch in the eye-in-hand image, (b) crops the switch and adds a pixel grid, and (c) repeats the crop after pressing. (d) Its difference image highlights changes in the lever and its shadow, providing evidence for checking whether the press took effect.
Figure 10: The agent combines image processing and geometric measurements to locate targets and check action outcomes. Panels (a–d) show a CaP-Bench Wipe episode before, during, and after a stroke; (e–h) show a RoboCasa365 kettle episode, including a gridded knob crop, projected burner centers, and detected flame. Panels (b), (f), (g), and (h) contain sensor observations processed by agent-written code.
| Tasks | Motion commands per episode | Through own programs | Feedback program | Reads force | Velocity servo |
| CaP-Bench, contact-rich (Nut, Wipe, Restack, 2A-Lift, Handover) | 14 | 99% | 67% | 86% | 4% |
| CaP-Bench, Lift and Stack | 9 | 86% | 44% | 97% | 6% |
| LIBERO-PRO, six perturbation cells | 20 | 68% | 37% | 64% | 3% |
| LIBERO-10 | 26 | 84% | 54% | 44% | 3% |
| RoboCasa365 | 50 | 86% | 71% | 63% | 12% |
Table 9: The agent uses both individual motion commands and programs with sensor-dependent control logic. Statistics cover all mainline episodes with Claude Code and Opus 5, grouped by task set. Motion commands is the median number of arm, gripper, or base commands per episode; Through own programs is the share issued by agent-written programs rather than starter tools. The remaining columns are shares of episodes: Feedback program uses a sensor reading to determine a motion, loop condition, or branch; Reads force reads the wrench topic; Velocity servo uses a feedback loop that streams Cartesian velocities.
| Tool | Purpose | CaP-Bench | LIBERO-PRO | RoboCasa365 |
| cam_snap | save an image topic to a file | 100% (7.9) | 100% (12.5) | 100% (17.8) |
| px2world | back-project a pixel with depth | 29% (0.6), read 24% | 75% (1.9), read 10% | 57% (1.6), read 24% |
| fjt_send | send one joint trajectory | 8% (0.1), read 33% | 63% (2.8), read 13% | 32% (2.0), read 30% |
| gripper_cmd | open / close the gripper | 9% (0.2), read 25% | 56% (1.8), read 17% | 22% (1.5), read 29% |
| ik_move | IK to a pose and move | 3% (0.1), read 41% | 41% (1.2), read 36% | 8% (0.6), read 50% |
| base_goto | drive the base to a pose | – | – | 62% (3.0), read 11% |
Table 10: Starter-tool use varies by operation and benchmark. For Claude Code with Opus 5 across all mainline episodes, each cell reports the percentage of episodes executing the tool, with mean calls per episode in parentheses. Where shown, read reports the percentage that only read the tool’s source.
Figure 11: The agent uses visual checkpoints to assess progress during a drawer task. Camera observations from the LIBERO-PRO episode in Figure 6 show (a) the initial scene, (b) the handle before grasping, (c) the drawer after pulling, (d) the reach toward the cream-cheese box, and (e) its release in the drawer.
Figure 12: Agent-written programs use position, gripper, and force readings to monitor and adjust motion. The traces are plotted from the programs’ printed logs. In the LIBERO-PRO drawer episode, (a) proportional velocity corrections bring the hand within 2.5 mm of the target (green band); (b) the pulling program checks displacement and finger gap after each burst, stopping if the gap falls below 8 mm (dashed line) or motion stalls. In CaP-Bench, (c) the Nut program reads the wrench after each of seven insertion segments, with the vertical load remaining at the tool’s weight and no lateral force observed; (d) the Wipe program checks contact before executing 153 position-controlled waypoints and reads the load again afterward.
Figure 13: The agent uses measured force, finger gap, orientation, and position error to guide motion. Traces plotted from program logs show (a) a force-guarded descent in Restack, (b) a staged two-arm lift, (c) handle leveling during handover, and (d) a velocity approach to a jar in RoboCasa365.
Figure 14: The agent writes programs that account for base motion and check grasps during transport. Code and terminal output from a RoboCasa365 hot-dog task show (a) driving before reaching, (b) correcting a move using forward kinematics, (c) checking finger gap during a staged lift, and (d) retrying a failed grasp.
Figure 15: Visual checkpoints reveal grasp failures during mobile manipulation. Agent-saved images from the RoboCasa365 episode in Figure 14 show the task’s progress; (c) and (d) are crops used to diagnose grasp problems before the agent completes both plates.
Figure 16: The agent writes code to coordinate both arms and verify each lift stage. Code and terminal output from CaP-Bench Two-Arm Lift show (a) phase-by-phase execution, (b) dispatching both trajectory goals before waiting, (c) a staged lift with a gripper-gap guard, and (d) the second attempt and its verification.
Figure 17: Front-camera checkpoints from the CaP-Bench Two-Arm Lift episode in Figure 16 , showing the failed first lift and successful second attempt.
| Agent | Model (Effort) | Success (%) | Turns, median (succ. / fail.) | Tokens / turn, median | Cost ($), mean | Cost per success ($) |
| CC | Sonnet 5 (high) | 24.0 | 301 (219 / 334) | 512 | 22.4 | 93.3 |
| CC | Opus 5 (high) | 72.5 | 103 (97 / 131) | 1,058 | 13.7 | 18.9 |
| CC | Fable 5.1 (high) | 75.0 | 73 (72 / 84) | 760 | 8.4 | 11.2 |
| Codex | GPT-6 Astra (medium) | 62.5 | 98 (103 / 89) | 90 | 11.8 | 19.1 |
| Codex | GPT-6 Astra (high) | 69.0 | 93 (97 / 86) | 122 | 10.0 | 14.4 |
Table 11: Higher success can coincide with fewer interactions and lower cost per success. Profiles of OpenRUA on the LIBERO-10 suite of LIBERO-PRO cover both perturbation cells, with 200 trials per configuration. Turns and output tokens per turn, including thinking, are medians over episodes; parenthesized turn counts separate successful and failed episodes. A turn is one model response. Cost is the mean per episode at API list prices, and cost per success is total cost divided by the number of successful episodes. Logging coverage and cost accounting are described below; Figure 18 shows the distributions.
Figure 18: Interaction and cost distributions differ across agent–model configurations and episode outcomes. Agent turns, output tokens per turn, and episode costs are shown separately for successful and failed runs on the LIBERO-10 suite of LIBERO-PRO. Costs use API list prices, as in Table 11 . Boxes span the interquartile range; whiskers extend to the most extreme values within 1.5 times that range.
| Class | Sub-cause | LIBERO | CaP | RoboCasa |
| A | Joint threshold: “open” needs mm or of travel, a mixer head ; lift needs m | 14 | 1 | 6 |
| Region box differs from the words: “in front of the stove”, “right of the plate”, “on the rack”, “in the microwave” | 19 | 0 | 0 | |
| Tolerance of On : body origins within 30 mm (a mug’s origin sits 18 mm toward its handle) | 2 | 0 | 0 | |
| Final check coarser than the predicate: drawer read as shut from a point cloud, door “closed” needs of travel, lid seated within 4 cm and | 7 | 0 | 10 | |
| Stand-off after release ( / m), contact and timer clauses, sweep m, “do not touch the food”, navigation pose within 0.20 m | 0 | 0 | 20 | |
| B | Predicate names the other object (“left moka pot” scores the pot in the BDDL’s right init region) | 9 | 0 | 0 |
Table 12: Detailed causes within the five failure categories. Counts are analyzed failed episodes with Claude Code and Opus 5, grouped as in Table 5 . Thresholds come from the benchmark predicates: LIBERO joint ranges and BDDL region boxes, and RoboCasa365 task success checks.
Figure 19: Failures can involve an unstated completion criterion, undocumented machine behavior, or an agent control error. Agent-saved images show one episode from each benchmark: (a) LIBERO wine-rack placement (class A), (b) RoboCasa365 freezer loading (class C), and (c) CaP-Bench handover (class E). Figures 21 , 22 and 23 show the corresponding checkpoint sequences; Figure 20 illustrates a class-B mismatch between instruction and predicate.
Figure 20: The benchmark scores a different moka pot from the one specified by the instruction. This class-B case comes from a LIBERO-PRO variant of “put the left moka pot on the stove” (nine of ten seeds; seed 2 shown). The BDDL file places moka_pot_2 in moka_pot_left_init_region and moka_pot_1 in moka_pot_right_init_region , but its goal is (On moka_pot_1 flat_stove_1_cook_region) . After checking the image-right direction of all seven cameras, the agent moves the pot from the left region and places it 1.2 mm from the burner center, with its full weight supported by the stove. The predicate instead checks the other pot.
Figure 21: A stable bottle placement misses the benchmark’s unstated target region. In this class-A LIBERO wine-rack case (seven of seven failed seeds; seed 5 shown), the agent models the shelf as a corrugated plane and lays the bottle along a groove to prevent rolling. Its recorded reasoning is “laying the bottle along a groove was the only stable option: cross-ways it rolls off”. The predicate checks the bottle’s base against wine_rack_1_top_region , a 22 mm band at the foot of the slope; the base rests 40 mm up the slope from that band and 10 mm outside its width. In the benchmark demonstrations, the bottle’s base rests in the band.
Figure 22: Repeatedly reopening a self-closing freezer door exhausts the episode budget. This class-C RoboCasa365 case shows seed 4 of “move the trays into the freezer”; all ten failed seeds reach the wall-clock limit. The door closes while the robot fetches a tray from the counter, forcing it to reopen the door on each return. The agent labels its move after (c) “Retract arm from the closed door”. The unbraked base also slides toward the fridge during pulling and can push the door shut. No dropped object or misread observation was identified as the decisive error.
Figure 23: An incorrect gripper command ejects the hammer during handover. This class-E CaP-Bench case shows seed 58; four of six failed seeds share this mechanism. The agent measures a 0.030 m handle but commands arm 1 to 0.008 m per finger, despite the workspace warning: “Command a target width, never 0 against an object; 0 keeps squeezing and can eject it”. When arm 0 releases, arm 1’s finger positions decrease from 0.0123 to 0.0010 m and the hammer falls to the floor. Earlier, arm 0 held the hammer 0.21 m from the head, allowing its angle to increase from to during transport.
Figure 24: Checking placement without checking object identity leads to a false completion. In this LIBERO-10 episode, the agent places the tomato can instead of the alphabet-soup can and reports success after inspecting the basket; (e) the requested blue can remains on the table.