HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents

Hongcheng Gao; Alex Zhou; Jay Zhu; Zelin Zheng; Shijia Ge; Xiping Li; Di Wu; Yazhe Wang

cs.ROOct 16, 2026

HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents

Authors: Hongcheng Gao, Alex Zhou, Jay Zhu, Zelin Zheng, Shijia Ge, Xiping Li, Di Wu, Yazhe Wang,

Abstract

Agents built on vision language models (VLMs) now compete with learned policies on robot manipulation leaderboards. We ask how far a coding agent can go when it controls a robot instead of a computer, and where it stops. Our system, HexaShell, keeps the harness thin: it only turns the agent's commands into robot motion. The agent drives a robot with two arms through one command line. It sees three RGB cameras and knows their calibration. After each failed attempt, the agent reads the log, finds the cause, and writes or fixes small programs called tools. In this way it keeps what it learns as code, as a coding agent does when it fixes a failing test. The tools may use only what the robot observes. They use triangulation to find where objects are, relative to each other and to the gripper. If two cameras see the same point, the point lies where the two camera rays meet. If a point lies on a known plane, such as the table or the top of a block, it lies where its ray meets that plane. When the head camera is not enough, the robot moves a wrist camera to get a second view. We build the tools on only ten layouts of one seed per task, then freeze them and test them on ten new layouts of the same seed. We use Codex with GPT-6-Astra at medium reasoning effort. On all 42 RoboDojo tasks, HexaShell reaches a success rate of 39.7% and a progress score of 48.0, averaged in the same way as the leaderboard. This is a new state of the art on RoboDojo. The limit is contact, not reasoning: pushing reaches 92.5% and memory tasks 65.0%, but tight insertion only 8.3%. When the agent fails, it usually sees the failure but cannot make the motion work. We open source the tools evolved so far at https://github.com/HexaFuture/HexaShell.

Figures & tables

Figure 1: HexaShell turns robot failures into reusable tools. A coding agent operates a robot with two arms through a thin harness and sees RGB images only. After a failure, an optimizer agent diagnoses the episode and writes or fixes tools that use only what the robot observes. Both agents are Codex with GPT-6-Astra at medium reasoning effort. Bottom: where tools help (rigid, planar, tool use, clutter) and where they fall short (cloth).

Figure 2: RoboDojo comparison on the leaderboard’s metrics: success rate (a) and progress score (b). Overall is the leaderboard’s score, the mean of its five task types (Generalization: mean of its standard and cluttered variants). HexaShell uses GPT-6-Astra at medium reasoning effort, RGB images only and frozen tools, tested on ten new layouts of each task, which the tools were not built on ( Section 4.1 ). GPT-6-Astra is the same model without tools [ 69 ] . Systems are ordered and shaded by Overall success rate.

Panel of Figure 3 — Figure 3: HexaShell. Top, one episode: \scriptsize1⃝ a coding agent in a sandbox issues robo commands; the server validates each one, turns it into joint motion and returns feedback and RGB observations. Bottom, between episodes: \scriptsize2⃝ the episode log keeps the commands, the camera frames and the agent’s transcript; \scriptsize3⃝ after a failure the optimizer agent diagnoses the episode, \scriptsize4⃝ writes or fixes a tool that must pass a delivery check, and \scriptsize5⃝ the tool is loaded and the same layout is tried again.

	Triangulation		Depth Anything		Rendered depth
Type	Succ.	Prog.	Succ.	Prog.	Succ.	Prog.
Generalization	34.2	37.7	26.7	30.0	45.4	51.5
Precision	21.3	28.4	7.5	11.9	41.3	47.3
Long-Horizon	18.8	32.2	13.8	29.2	32.5	44.2
Memory	73.3	73.3	55.0	58.0	66.6	66.6
Open	43.8	46.6	45.0	48.1	58.5	60.7

Table 1: Triangulation against a depth network, same harness, loop and model. Success rate and progress score in %. Triangulation is our RGB tool set; Depth Anything V2 estimates depth from each image. Both are frozen retests on the development layouts. Rendered depth is not part of the standard observation and is shown for reference. Last row: layouts that only one of the two passes, of 420.

	Triangulation	Depth network
Executed steps spent on viewing	3.6%	0.7%
Episodes with a viewing move	180 (43%)	40 (10%)
viewing steps per step limit, median	4.9%	2.9%
Successes	152	118
step limit used, median	70.0%	63.1%
Failures that used 75% of the limit or more	189 / 268	228 / 302

Table 2: What second views cost, 420 retest episodes each on the development layouts. A viewing move is a base motion followed, before any other motion, by a measurement that reads a wrist camera, or one of three tools whose only job is to look. Viewing inside larger tools is not counted, so the first three rows are lower bounds.

		Success %					Failed episodes, %
Class	Var.	None	DN	Ours	VPP2	At 0	Decl.	Exh.	Gave up
R rigid pick, place, stack	22	36.9	44.1	50.0	30.0	3	29	60	9
Pu push, slide, press	4	60.0	67.5	92.5	24.7	0	33	33	33
Mv moving objects	2	33.0	30.0	40.0	43.0	0	46	15	38
In tight insertion	12	1.7	1.7	8.3	28.1	7	32	53	15
Df deformable, fluid	6	26.0	30.0	33.3	49.4	2	15	77	8

Table 3: Capability map and failure modes. Mean success over variants for the same model without tools (None, [ 69 ] ), HexaShell with a depth network (DN, development layouts) and with triangulation (Ours, new layouts), and VPP2, the highest learned policy; RGB variants at 0% on the new layouts; and how failed RGB episodes on the development layouts ended: the agent declared completion and the checker disagreed, the budget was exhausted (less than a quarter left), or the agent gave up with more left.

Task	Layout	Class	Fails	Found	Changed	Edit	Min.	Next	Retest
Push T	2	Pu	2	area check counted grid cells, so cameras with different pixel sizes disagreed; height refused	area from each pixel’s footprint on the plane	+49/ $-$ 11	7	354/600	pass	10/10
Swap T	0	R, M	2	colour mask mixed top and side faces; height failed seven times	match colour by hue, keep the bright top face	+75/ $-$ 5	8	285/400	pass	10/10
Play Xylophone	1	Ar	4	one camera put the held mallet head 23 mm off	match a second view and triangulate	+109/ $-$ 7	5	377/500	pass	6/10
Match and pick from conveyor	4	Mv	1	pick schedules shorter than the arm’s motion; two aborts	estimate motion cost first, close later	+132/ $-$ 5	6	587/700	pass	7/10
Play tic tac toe	0	R, S	3	last ring needed 146 steps, 84 left	turn the gripper while moving	+194/ $-$ 5	11	1028/1100	pass	9/10
Fold clothes, cluttered	1	Df	1	grasp point 1.5 mm from the cloth edge	require 12 mm of cloth around the point	+112/ $-$ 12	9	344/500	fail	6/10

Table 4: Seven turns of the loop. Each row is one layout: the attempts that failed on it before the success (Fails), what the optimizer found after the last failure, what it changed, the size of the edit in lines added and removed (tests included) and the length of the round in minutes, the action steps of the next attempt, which succeeded, and the frozen retest of the same layout, with the task’s retest over its ten development layouts.

Figure 4: Push T, layout 2: a height that the tool refused. Left wrist camera. (a) and (c) show almost the same placement; only the measured height of the block’s top differs.

Figure 5: Swap T, layout 0: two faces of one colour. (a, c) left wrist camera; (b, d) head camera, cropped.

Figure 6: Play Xylophone, layout 1: one view against two. (a) head camera, cropped; (b–d) right wrist camera.

Figure 7: Match and pick from conveyor, layout 4: time, not position. (a, c) right wrist camera; (b) head camera, cropped. (c) is the last frame of the video, since the episode ended during the lift.

Figure 8: Play tic tac toe, layout 0: the step budget. (a, b) head camera, cropped, before the last ring; (c) right wrist camera, last frame of the video.

Figure 9: Fold clothes, cluttered layout 1: where to grasp a sleeve. Head camera, cropped, after the sleeve folds (a, c) and after the repair grasp of round 13 (b) and the crossed sleeves of round 14 (d).

Figure 10: Insert tubes, layout 4: a stop after the lift. (a, b, d) head camera, cropped; (c) left wrist camera.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Task	Phys.	Dec.	RGB	Depth net.	Rendered %	VPP2 %	PhysicalRSI %
Generalization	arrange largest number	R	S	1/5 / 1/5	3/5 / 2/5	80 / 40	4 / 0	1.3 / 21.3
	fold clothes §	Df	G	4/5 / 1/5	2/5 / 1/5	30.7 / 18.7 ⋆	74.7 / 61.3	41.3 / 34.7
	hang mugs §	In	G	0/5 / 0/5	0/5 / 0/5	40 / 0	64 / 0	0 / 0
	make toast §	Ar	G	0/5 / 0/5	0/5 / 0/5	0 / 0	56 / 22.7	0 / 0
	pack objects into box §	R	S	0/5 / 0/5	0/5 / 0/5	0 / 40	10.7 / 2.7	5.3 / 0
	pour liquid into cup §	Df	G	0/5 / 0/5	1/5 / 2/5	94.7 / 58.7 ⋆†	68 / 32	93.3 / 9.3

Appendix

Table 5: Results on all 42 RoboDojo tasks. Phys. and Dec.: the physical interaction and the decision a task requires ( Section 5.1 ): R rigid pick, place or stack; Pu push, slide or press; In tight insertion; Ar articulated object or tool use; Df deformable, fluid or granular; Mv moving objects; G geometry only; S symbols, language or categories; M memory. RGB: frozen retests on ten new layouts, 10 to 19 of seed 0 (Generalization: standard / cluttered 5 to 9); swap blocks with the benchmark’s fix of its button reset and tools developed again under it. Depth net.: frozen retests on the ten seed-0 development layouts (Generalization: five standard / five cluttered). § The optimizer read a note on what the checker requires ( Section 3.4 ). Rendered: success in % of the earlier tools with rendered depth; ⋆ 50 episodes on each of seeds 0–2 (25 per variant for Generalization tasks), otherwise the development layouts; † tools from a run with checker notes, ‡ tools rebuilt by a team member ( Appendix D ). VPP2, PhysicalRSI: leaderboard success in %.

Figure 11: Success rate on every task variant: HexaShell against VPP2 and PhysicalRSI. Variants are sorted by HexaShell’s success rate; rand: cluttered layouts. HexaShell: RGB images only, frozen retest on ten new layouts, 10 to 19 of seed 0; VPP2 and PhysicalRSI: leaderboard rollouts, 50 episodes per seed.

Figure 12: Progress score per task variant on the development layouts, HexaShell with triangulation against (a) the depth network version and (b) the same model without tools. Points above the diagonal favour triangulation. Colour and shape give the physical class ( Section 5.1 ): rigid and push (R, Pu), insertion (In), and articulated, deformable or moving (Ar, Df, Mv). Means over the 54 variants: 40.8, 33.0 and 29.8.

Type	Task	Commands	Lines	Rounds	View	Retest
Generalization	arrange largest number	7	1,785	23	0.9	3/10
	fold clothes	7	2,310	21	0.0	6/10
	hang mugs	13	2,944	12	12.4	0/10
	make toast	15	2,022	11	4.5	1/10
	pack objects into box	7	1,758	16	0.2	0/10
	pour liquid into cup	4	2,548	31	20.3	1/10

Appendix

Table 6: RGB tools per task. Commands: task commands besides the shared tools; Lines: lines of Python in the enabled tools and the modules they import; Rounds: optimizer rounds of the RGB run; View: executed steps spent on viewing moves in the retest, in %; Retest: successes on the ten development layouts.

Task	Round 1	Round 2	%	What the note supplied
press by number	0/10	10/10, 10/10	100	confirm after each group; no extra press
pour liquid into cup	0/5 / 2/5	5/5 / 5/5	94.7 / 58.7	judged once the bottle is upright again
pour balls into vase	2/10	10/10, 9/10	82.7	three end conditions; reach failures
deposit coin	3/10	6/10	29.3	coin must fall through; wrist roll drops it
fold clothes	3/5 / 0/5	2/10, 4/10	(30.7 / 18.7)	keypoint windows and fold order
make kong	0/10	8/10	11.0	a second, unstated stage

Appendix

Table 7: Run with rendered depth and checker notes: what a note changes. Retests on the ten development layouts (standard / random for Generalization tasks) and success in % over seeds 0–2 (fold clothes, in parentheses: the tools of the first round, which were kept).

Agentic AI for Robotics

See all 39 papers

Sep 29, 2026

RoboHarness: A Simple Harness Outperforms VLA and World Action Models

The rise of large language models (LLMs) has opened a path toward generalist embodied AI, raising the question of how LLMs can act in the physical world. One straightforward approach is training the model to generate actions: collect robot-action data and video demonstration, refine LLM architecture with an action head, and train a vision-language-action (VLA) model at scale. However, we argue that coding agents have the potential to realize generalist embodied manipulation. All they need is a simple yet effective robot harness. We present RoboHarness: a robotic harness that gives LLM agents a visual-geometric control panel so that they can directly understand and invoke embodied tasks. With RoboHarness, Qwen3.8-Flash-Next outperforms state-of-the-art VLA models on BEHAVIOR Challenge 2025 tasks. The implementation is open source.

Bince Qu, Wei Chen, Bo Zhang

Zhejiang University

Agentic AI for Robotics LLM Agent Harnesses Robotic Control Tool-Using Vision-Language Agents

Ask PDF

Jun 15, 2026

RHO: Your Coding Agent is Secretly a Roboticist

Code-as-Policies (CaP) has shown that large language models (LLMs) can write code to solve robotics tasks by composing perception, planning, and control primitives. Recent CaP systems, however, rely on multi-turn code-generation loops at test time, which is often infeasible for real-time robot control. We introduce Robotics Harness Optimization (RHO), a novel paradigm in which tool-enabled coding agents, at training time, propose and search for interpretable, neurosymbolic multi-file policy repositories (Repositories-as-Policies) that compose these primitives rather than a single prompt, function, or file. RHO searches with reflective feedback from environment reward and execution rather than teleoperation demonstrations. It generalizes to perturbed pick-and-place settings like LIBERO-PRO, where OpenVLA scores 0.0% and

π_{0.5}

averages 12.83%. Using the same low-level primitives, RHO reaches a 45.0% success rate, 2.5x higher than the strongest multi-turn agentic system, and 3.5x higher than

π_{0.5}

. On Robosuite, RHO sets a new state-of-the-art of 70.0%, exceeding the prior multi-turn record of 68.29% using single-turn execution with no corrective LLM code edits at deployment. When an LLM is used in the control loop, as on RAI's O3DE benchmark, RHO optimizes the deployed agent's multi-file harness of prompts, tools, and control code, improving held-out success from 23.5% to 44.3% with 20% less wall-clock time and 27% fewer tool calls.

Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade +3

University of California, Berkeley · Embodied Science · AMD

LLM Agent Harnesses Robotic Manipulation Robot Policy Learning AI Coding Agents Large Language Model-Based Robot Planning Program Synthesis+4

Ask PDF

Sep 9, 2026

Show-Harness: Just a VLM Agent Can Play Robots

Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.

Yanzhe Chen, Zechen Bai, Zhijun Cao +7

Show Lab, National University of Singapore

Cross-Embodiment Robot Learning Language-Conditioned Robot Control VLMs for Robotics Vision-Language Agents Agentic AI for Robotics

Ask PDF

Abstract

Agentic AI for Robotics

Explore similar work

RoboHarness: A Simple Harness Outperforms VLA and World Action Models

RHO: Your Coding Agent is Secretly a Roboticist

Show-Harness: Just a VLM Agent Can Play Robots