cs.ROOct 16, 2026
SaveHexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents
Abstract
Agents built on vision language models (VLMs) now compete with learned policies on robot manipulation leaderboards. We ask how far a coding agent can go when it controls a robot instead of a computer, and where it stops. Our system, HexaShell, keeps the harness thin: it only turns the agent's commands into robot motion. The agent drives a robot with two arms through one command line. It sees three RGB cameras and knows their calibration. After each failed attempt, the agent reads the log, finds the cause, and writes or fixes small programs called tools. In this way it keeps what it learns as code, as a coding agent does when it fixes a failing test. The tools may use only what the robot observes. They use triangulation to find where objects are, relative to each other and to the gripper. If two cameras see the same point, the point lies where the two camera rays meet. If a point lies on a known plane, such as the table or the top of a block, it lies where its ray meets that plane. When the head camera is not enough, the robot moves a wrist camera to get a second view. We build the tools on only ten layouts of one seed per task, then freeze them and test them on ten new layouts of the same seed. We use Codex with GPT-6-Astra at medium reasoning effort. On all 42 RoboDojo tasks, HexaShell reaches a success rate of 39.7% and a progress score of 48.0, averaged in the same way as the leaderboard. This is a new state of the art on RoboDojo. The limit is contact, not reasoning: pushing reaches 92.5% and memory tasks 65.0%, but tight insertion only 8.3%. When the agent fails, it usually sees the failure but cannot make the motion work. We open source the tools evolved so far at https://github.com/HexaFuture/HexaShell.
Figures & tables
Figure 1: HexaShell turns robot failures into reusable tools. A coding agent operates a robot with two arms through a thin harness and sees RGB images only. After a failure, an optimizer agent diagnoses the episode and writes or fixes tools that use only what the robot observes. Both agents are Codex with GPT-6-Astra at medium reasoning effort. Bottom: where tools help (rigid, planar, tool use, clutter) and where they fall short (cloth).
Figure 2: RoboDojo comparison on the leaderboard’s metrics: success rate (a) and progress score (b). Overall is the leaderboard’s score, the mean of its five task types (Generalization: mean of its standard and cluttered variants). HexaShell uses GPT-6-Astra at medium reasoning effort, RGB images only and frozen tools, tested on ten new layouts of each task, which the tools were not built on ( Section 4.1 ). GPT-6-Astra is the same model without tools [ 69 ] . Systems are ordered and shaded by Overall success rate.
Figure 3: HexaShell. Top, one episode: \scriptsize1⃝ a coding agent in a sandbox issues robo commands; the server validates each one, turns it into joint motion and returns feedback and RGB observations. Bottom, between episodes: \scriptsize2⃝ the episode log keeps the commands, the camera frames and the agent’s transcript; \scriptsize3⃝ after a failure the optimizer agent diagnoses the episode, \scriptsize4⃝ writes or fixes a tool that must pass a delivery check, and \scriptsize5⃝ the tool is loaded and the same layout is tried again.
| Triangulation | Depth Anything | Rendered depth | ||||
| Type | Succ. | Prog. | Succ. | Prog. | Succ. | Prog. |
| Generalization | 34.2 | 37.7 | 26.7 | 30.0 | 45.4 | 51.5 |
| Precision | 21.3 | 28.4 | 7.5 | 11.9 | 41.3 | 47.3 |
| Long-Horizon | 18.8 | 32.2 | 13.8 | 29.2 | 32.5 | 44.2 |
| Memory | 73.3 | 73.3 | 55.0 | 58.0 | 66.6 | 66.6 |
| Open | 43.8 | 46.6 | 45.0 | 48.1 | 58.5 | 60.7 |
Table 1: Triangulation against a depth network, same harness, loop and model. Success rate and progress score in %. Triangulation is our RGB tool set; Depth Anything V2 estimates depth from each image. Both are frozen retests on the development layouts. Rendered depth is not part of the standard observation and is shown for reference. Last row: layouts that only one of the two passes, of 420.
| Triangulation | Depth network | |
| Executed steps spent on viewing | 3.6% | 0.7% |
| Episodes with a viewing move | 180 (43%) | 40 (10%) |
| viewing steps per step limit, median | 4.9% | 2.9% |
| Successes | 152 | 118 |
| step limit used, median | 70.0% | 63.1% |
| Failures that used 75% of the limit or more | 189 / 268 | 228 / 302 |
Table 2: What second views cost, 420 retest episodes each on the development layouts. A viewing move is a base motion followed, before any other motion, by a measurement that reads a wrist camera, or one of three tools whose only job is to look. Viewing inside larger tools is not counted, so the first three rows are lower bounds.
| Success % | Failed episodes, % | ||||||||
| Class | Var. | None | DN | Ours | VPP2 | At 0 | Decl. | Exh. | Gave up |
| R rigid pick, place, stack | 22 | 36.9 | 44.1 | 50.0 | 30.0 | 3 | 29 | 60 | 9 |
| Pu push, slide, press | 4 | 60.0 | 67.5 | 92.5 | 24.7 | 0 | 33 | 33 | 33 |
| Mv moving objects | 2 | 33.0 | 30.0 | 40.0 | 43.0 | 0 | 46 | 15 | 38 |
| In tight insertion | 12 | 1.7 | 1.7 | 8.3 | 28.1 | 7 | 32 | 53 | 15 |
| Df deformable, fluid | 6 | 26.0 | 30.0 | 33.3 | 49.4 | 2 | 15 | 77 | 8 |
Table 3: Capability map and failure modes. Mean success over variants for the same model without tools (None, [ 69 ] ), HexaShell with a depth network (DN, development layouts) and with triangulation (Ours, new layouts), and VPP2, the highest learned policy; RGB variants at 0% on the new layouts; and how failed RGB episodes on the development layouts ended: the agent declared completion and the checker disagreed, the budget was exhausted (less than a quarter left), or the agent gave up with more left.
| Task | Layout | Class | Fails | Found | Changed | Edit | Min. | Next | Retest | |
|---|---|---|---|---|---|---|---|---|---|---|
| Push T | 2 | Pu | 2 | area check counted grid cells, so cameras with different pixel sizes disagreed; height refused | area from each pixel’s footprint on the plane | +49/ 11 | 7 | 354/600 | pass | 10/10 |
| Swap T | 0 | R, M | 2 | colour mask mixed top and side faces; height failed seven times | match colour by hue, keep the bright top face | +75/ 5 | 8 | 285/400 | pass | 10/10 |
| Play Xylophone | 1 | Ar | 4 | one camera put the held mallet head 23 mm off | match a second view and triangulate | +109/ 7 | 5 | 377/500 | pass | 6/10 |
| Match and pick from conveyor | 4 | Mv | 1 | pick schedules shorter than the arm’s motion; two aborts | estimate motion cost first, close later | +132/ 5 | 6 | 587/700 | pass | 7/10 |
| Play tic tac toe | 0 | R, S | 3 | last ring needed 146 steps, 84 left | turn the gripper while moving | +194/ 5 | 11 | 1028/1100 | pass | 9/10 |
| Fold clothes, cluttered | 1 | Df | 1 | grasp point 1.5 mm from the cloth edge | require 12 mm of cloth around the point | +112/ 12 | 9 | 344/500 | fail | 6/10 |
Table 4: Seven turns of the loop. Each row is one layout: the attempts that failed on it before the success (Fails), what the optimizer found after the last failure, what it changed, the size of the edit in lines added and removed (tests included) and the length of the round in minutes, the action steps of the next attempt, which succeeded, and the frozen retest of the same layout, with the task’s retest over its ten development layouts.
Figure 4: Push T, layout 2: a height that the tool refused. Left wrist camera. (a) and (c) show almost the same placement; only the measured height of the block’s top differs.
Figure 5: Swap T, layout 0: two faces of one colour. (a, c) left wrist camera; (b, d) head camera, cropped.
Figure 6: Play Xylophone, layout 1: one view against two. (a) head camera, cropped; (b–d) right wrist camera.
Figure 7: Match and pick from conveyor, layout 4: time, not position. (a, c) right wrist camera; (b) head camera, cropped. (c) is the last frame of the video, since the episode ended during the lift.
Figure 8: Play tic tac toe, layout 0: the step budget. (a, b) head camera, cropped, before the last ring; (c) right wrist camera, last frame of the video.
Figure 9: Fold clothes, cluttered layout 1: where to grasp a sleeve. Head camera, cropped, after the sleeve folds (a, c) and after the repair grasp of round 13 (b) and the crossed sleeves of round 14 (d).
Figure 10: Insert tubes, layout 4: a stop after the lift. (a, b, d) head camera, cropped; (c) left wrist camera.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Type | Task | Phys. | Dec. | RGB | Depth net. | Rendered % | VPP2 % | PhysicalRSI % |
| Generalization | arrange largest number | R | S | 1/5 / 1/5 | 3/5 / 2/5 | 80 / 40 | 4 / 0 | 1.3 / 21.3 |
| fold clothes § | Df | G | 4/5 / 1/5 | 2/5 / 1/5 | 30.7 / 18.7 ⋆ | 74.7 / 61.3 | 41.3 / 34.7 | |
| hang mugs § | In | G | 0/5 / 0/5 | 0/5 / 0/5 | 40 / 0 | 64 / 0 | 0 / 0 | |
| make toast § | Ar | G | 0/5 / 0/5 | 0/5 / 0/5 | 0 / 0 | 56 / 22.7 | 0 / 0 | |
| pack objects into box § | R | S | 0/5 / 0/5 | 0/5 / 0/5 | 0 / 40 | 10.7 / 2.7 | 5.3 / 0 | |
| pour liquid into cup § | Df | G | 0/5 / 0/5 | 1/5 / 2/5 | 94.7 / 58.7 ⋆† | 68 / 32 | 93.3 / 9.3 |
Table 5: Results on all 42 RoboDojo tasks. Phys. and Dec.: the physical interaction and the decision a task requires ( Section 5.1 ): R rigid pick, place or stack; Pu push, slide or press; In tight insertion; Ar articulated object or tool use; Df deformable, fluid or granular; Mv moving objects; G geometry only; S symbols, language or categories; M memory. RGB: frozen retests on ten new layouts, 10 to 19 of seed 0 (Generalization: standard / cluttered 5 to 9); swap blocks with the benchmark’s fix of its button reset and tools developed again under it. Depth net.: frozen retests on the ten seed-0 development layouts (Generalization: five standard / five cluttered). § The optimizer read a note on what the checker requires ( Section 3.4 ). Rendered: success in % of the earlier tools with rendered depth; ⋆ 50 episodes on each of seeds 0–2 (25 per variant for Generalization tasks), otherwise the development layouts; † tools from a run with checker notes, ‡ tools rebuilt by a team member ( Appendix D ). VPP2, PhysicalRSI: leaderboard success in %.
Figure 11: Success rate on every task variant: HexaShell against VPP2 and PhysicalRSI. Variants are sorted by HexaShell’s success rate; rand: cluttered layouts. HexaShell: RGB images only, frozen retest on ten new layouts, 10 to 19 of seed 0; VPP2 and PhysicalRSI: leaderboard rollouts, 50 episodes per seed.
Figure 12: Progress score per task variant on the development layouts, HexaShell with triangulation against (a) the depth network version and (b) the same model without tools. Points above the diagonal favour triangulation. Colour and shape give the physical class ( Section 5.1 ): rigid and push (R, Pu), insertion (In), and articulated, deformable or moving (Ar, Df, Mv). Means over the 54 variants: 40.8, 33.0 and 29.8.
| Type | Task | Commands | Lines | Rounds | View | Retest |
| Generalization | arrange largest number | 7 | 1,785 | 23 | 0.9 | 3/10 |
| fold clothes | 7 | 2,310 | 21 | 0.0 | 6/10 | |
| hang mugs | 13 | 2,944 | 12 | 12.4 | 0/10 | |
| make toast | 15 | 2,022 | 11 | 4.5 | 1/10 | |
| pack objects into box | 7 | 1,758 | 16 | 0.2 | 0/10 | |
| pour liquid into cup | 4 | 2,548 | 31 | 20.3 | 1/10 |
Table 6: RGB tools per task. Commands: task commands besides the shared tools; Lines: lines of Python in the enabled tools and the modules they import; Rounds: optimizer rounds of the RGB run; View: executed steps spent on viewing moves in the retest, in %; Retest: successes on the ten development layouts.
| Task | Round 1 | Round 2 | % | What the note supplied |
|---|---|---|---|---|
| press by number | 0/10 | 10/10, 10/10 | 100 | confirm after each group; no extra press |
| pour liquid into cup | 0/5 / 2/5 | 5/5 / 5/5 | 94.7 / 58.7 | judged once the bottle is upright again |
| pour balls into vase | 2/10 | 10/10, 9/10 | 82.7 | three end conditions; reach failures |
| deposit coin | 3/10 | 6/10 | 29.3 | coin must fall through; wrist roll drops it |
| fold clothes | 3/5 / 0/5 | 2/10, 4/10 | (30.7 / 18.7) | keypoint windows and fold order |
| make kong | 0/10 | 8/10 | 11.0 | a second, unstated stage |
Table 7: Run with rendered depth and checker notes: what a note changes. Retests on the ten development layouts (standard / random for Generalization tasks) and success in % over seeds 0–2 (fold clothes, in parentheses: the tools of the first round, which were kept).