cs.ROSep 30, 2026

Scale and Selection: What Makes Automatic Harness Evolution Work for Visual-Interface Robot Agents

Authors: Zhijie Wei, Ferris Tan, Jinghui Wang

Organizations: Novaxbot

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When an off-the-shelf coding agent is used directly as a robot policy, observing a browser-based 3D interface through screenshots and acting by posing a virtual target gripper through a few tools, the agent's harness, its prompts, tools, and control rules, largely determines success, and until now it has been written by hand. We show that this harness can be improved automatically by another coding agent, the optimizer agent, and report two findings about what makes it work. First, the number of rollouts the optimizer agent sees per round governs whether the evolved harness is trustworthy, generalizes, and improves steadily. A single rollout is a noisy binary outcome, so with few rollouts per round a revision can be promoted on luck; enlarging the batch raises the signal-to-noise ratio of every promotion decision. Holding rounds fixed and growing the training set from 5 to 100 rollouts, held-out success rises from 47% to 67%, while small training sets overfit, reaching 70% on training tasks but only 54% held-out. Second, the optimizer agent must not be given free rein. With every revision it proposes accepted unconditionally, performance drifts downward within ten rounds as ill-judged edits accumulate; adding the most basic safeguard, Champion-Challenger selection that promotes a revision only if it strictly beats the incumbent on the same fixed evaluation set, turns the same loop into one that raises held-out success from 51% to 67% over 30 rounds. Automatic harness evolution for visual-interface robot agents is thus feasible, but its gains hinge on the rollout scale behind each decision and on how the optimizer agent's revisions are selected.

Figures & tables

Figure 1: Overview. (a) The task agent is an unmodified coding agent that controls a robot through a visual interface; the agent’s harness is what we evolve. (b) Each round, an optimizer agent reads the full trajectories of BB rollouts and proposes one bounded revision; under Champion–Challenger selection the revision is promoted only if it strictly beats the incumbent on the same fixed cases, whereas accepting every revision drifts downward within ten rounds (inset: training success at B=100B=100 ). (c) Held-out success of the evolved harness rises with the number of rollouts per round, from 47% at B=5B=5 to 67% at B=100B=100 .
Figure 2: Training batch size. (a) Held-out success of the final champion of each 30-round campaign (blue) and the mean and range over the five held-out nodes of each campaign (grey), against the batch size BB ; the dotted line is the baseline harness. (b) Training success versus held-out success for the baseline and 20 evolved harnesses, coloured by BB ; points above the diagonal generalize beyond their training score. The B=10B=10 champion with the highest training score of all campaigns, 70%, reaches only 54% held-out.
BBtraining successpromotionsheld-out successheld-out success
 baseline →\rightarrow final champion(of 30)final champion5-node mean (range)
520% →\rightarrow 40% (R3)147%44.6% (35–49)
1030% →\rightarrow 70% (R27)354%51.4% (48–54)
2040% →\rightarrow 55% (R28)251%52.8% (50–55)
10041% →\rightarrow 60% (R29)867%61.0% (51–67)
Baseline harness on the held-out set: 51%.
Table 1: Independent 30-round Champion–Challenger campaigns from the same baseline harness, differing only in the training batch size BB . Held-out numbers are on 100 rollouts from tasks disjoint from training.
Figure 3: Selection rule. Training success per round at each batch size under unconditional acceptance (grey; every revision becomes the next base) and Champion–Challenger selection (blue; the champion trajectory, with promoted challengers as triangles and rejected challengers as crosses). The dotted line is the baseline score. The unconditional runs were stopped after 10 rounds ( B=100B=100 ) or 20 rounds ( B=5,10,20B=5,10,20 ) once the downward trend was clear; their baseline commit was evaluated separately, so starting points differ slightly from the Champion–Challenger baselines.
Figure 4: Before and after harness evolution on RoboCasa. Two kitchen tasks with the same fixed seed, shown from the same camera: the initial scene, the final frame under the baseline harness, and the final frame under the round-2 champion. With the baseline the drawer stays shut and the bottle stays on the cabinet shelf; with the evolved harness the drawer is pulled open and the bottle is placed on the counter.
roundharnesstraining successdecision
0baseline harness2/100—
1challenger26/100promoted
2challenger37/100promoted (final champion)
3challenger28/100rejected
4challenger33/100rejected
Table 2: Champion–Challenger campaign on ten RoboCasa kitchen tasks with B=100B=100 .
 baseline harness H0H_{0}evolved harness (round 2)
task agentGPT-5.6 SolGPT-6 AstraGPT-5.6 SolGPT-6 Astra
successes / 1002373783
Table 3: Harness ×\times model on the ten RoboCasa tasks. The harness was evolved with GPT-5.6 Sol only; GPT-6 Astra was never used during evolution. Successes on the same 100 cases (ten tasks, ten seeds each).

Explore similar work

CardsList

  1. Rethinking the Evaluation of Harness Evolution for Agents

    Jul 14, 2026Yike Wang, Huaisheng Zhu, Zhengyu Hu +8Large Language Model AgentsTest-Time Scaling

    Allen Institute for AI · University of Washington · Independent

  2. HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

    Aug 3, 2026Luan Zhang, Ruochen Zhou, Dandan Song +9Evolving HarnessAgent Harness

    Beijing Institute of Technology, China · City University of Hong Kong, China · Independent, China

  3. EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?

    Date pendingZixuan Ke, Vaidehi Patil, Haizhou Shi +9Evolving HarnessAgentic Benchmarks

    Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison +1