cs.ROSep 30, 2026

Scale and Selection: What Makes Automatic Harness Evolution Work for Visual-Interface Robot Agents

Authors: Zhijie Wei, Ferris Tan, Jinghui Wang

Organizations: Novaxbot

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When an off-the-shelf coding agent is used directly as a robot policy, observing a browser-based 3D interface through screenshots and acting by posing a virtual target gripper through a few tools, the agent's harness, its prompts, tools, and control rules, largely determines success, and until now it has been written by hand. We show that this harness can be improved automatically by another coding agent, the optimizer agent, and report two findings about what makes it work. First, the number of rollouts the optimizer agent sees per round governs whether the evolved harness is trustworthy, generalizes, and improves steadily. A single rollout is a noisy binary outcome, so with few rollouts per round a revision can be promoted on luck; enlarging the batch raises the signal-to-noise ratio of every promotion decision. Holding rounds fixed and growing the training set from 5 to 100 rollouts, held-out success rises from 47% to 67%, while small training sets overfit, reaching 70% on training tasks but only 54% held-out. Second, the optimizer agent must not be given free rein. With every revision it proposes accepted unconditionally, performance drifts downward within ten rounds as ill-judged edits accumulate; adding the most basic safeguard, Champion-Challenger selection that promotes a revision only if it strictly beats the incumbent on the same fixed evaluation set, turns the same loop into one that raises held-out success from 51% to 67% over 30 rounds. Automatic harness evolution for visual-interface robot agents is thus feasible, but its gains hinge on the rollout scale behind each decision and on how the optimizer agent's revisions are selected.

Figures & tables

Figure 1: Overview. (a) The task agent is an unmodified coding agent that controls a robot through a visual interface; the agent’s harness is what we evolve. (b) Each round, an optimizer agent reads the full trajectories of BB rollouts and proposes one bounded revision; under Champion–Challenger selection the revision is promoted only if it strictly beats the incumbent on the same fixed cases, whereas accepting every revision drifts downward within ten rounds (inset: training success at B=100B=100 ). (c) Held-out success of the evolved harness rises with the number of rollouts per round, from 47% at B=5B=5 to 67% at B=100B=100 .
Figure 2: Training batch size. (a) Held-out success of the final champion of each 30-round campaign (blue) and the mean and range over the five held-out nodes of each campaign (grey), against the batch size BB ; the dotted line is the baseline harness. (b) Training success versus held-out success for the baseline and 20 evolved harnesses, coloured by BB ; points above the diagonal generalize beyond their training score. The B=10B=10 champion with the highest training score of all campaigns, 70%, reaches only 54% held-out.
BBtraining successpromotionsheld-out successheld-out success
 baseline →\rightarrow final champion(of 30)final champion5-node mean (range)
520% →\rightarrow 40% (R3)147%44.6% (35–49)
1030% →\rightarrow 70% (R27)354%51.4% (48–54)
2040% →\rightarrow 55% (R28)251%52.8% (50–55)
10041% →\rightarrow 60% (R29)867%61.0% (51–67)
Baseline harness on the held-out set: 51%.
Table 1: Independent 30-round Champion–Challenger campaigns from the same baseline harness, differing only in the training batch size BB . Held-out numbers are on 100 rollouts from tasks disjoint from training.
Figure 3: Selection rule. Training success per round at each batch size under unconditional acceptance (grey; every revision becomes the next base) and Champion–Challenger selection (blue; the champion trajectory, with promoted challengers as triangles and rejected challengers as crosses). The dotted line is the baseline score. The unconditional runs were stopped after 10 rounds ( B=100B=100 ) or 20 rounds ( B=5,10,20B=5,10,20 ) once the downward trend was clear; their baseline commit was evaluated separately, so starting points differ slightly from the Champion–Challenger baselines.
Figure 4: Before and after harness evolution on RoboCasa. Two kitchen tasks with the same fixed seed, shown from the same camera: the initial scene, the final frame under the baseline harness, and the final frame under the round-2 champion. With the baseline the drawer stays shut and the bottle stays on the cabinet shelf; with the evolved harness the drawer is pulled open and the bottle is placed on the counter.
roundharnesstraining successdecision
0baseline harness2/100—
1challenger26/100promoted
2challenger37/100promoted (final champion)
3challenger28/100rejected
4challenger33/100rejected
Table 2: Champion–Challenger campaign on ten RoboCasa kitchen tasks with B=100B=100 .
 baseline harness H0H_{0}evolved harness (round 2)
task agentGPT-5.6 SolGPT-6 AstraGPT-5.6 SolGPT-6 Astra
successes / 1002373783
Table 3: Harness ×\times model on the ten RoboCasa tasks. The harness was evolved with GPT-5.6 Sol only; GPT-6 Astra was never used during evolution. Successes on the same 100 cases (ten tasks, ten seeds each).

Explore similar work

CardsList
Jul 14, 2026cs.AI

Rethinking the Evaluation of Harness Evolution for Agents

Harness evolution is an iterative search procedure that repeatedly evaluates and revises candidate harnesses used for LLM agents using task feedback. We revisit the evaluation of such automatic harness evolution procedures and identify two fundamental issues in the protocol. First, prior work does not compare these approaches with simple task-level search baselines under matched feedback and inference budgets. Second, prior work searches for harness configurations using verification signals (e.g., unit test cases) drawn from the same benchmarks on which it reports the final performance of the evolved harnesses, violating the standard separation between training and test data. To address this, we compare automatic harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets, and evaluate evolved harnesses on held-out tasks to assess generalization. Following prior work, we experiment on Terminal-Bench 2.1 and find that automatic harness evolution fails to outperform simple test-time scaling methods both with and without test cases, and exhibits limited generalization. However, we find that long-horizon games are a promising setting for automatic harness evolution, as they are difficult enough to leave headroom, rely heavily on adaptation to out-of-distribution dynamics, and provide granular feedback by design. In these settings, task-specific harness evolution improves over the search baseline by 80% on ARC-AGI-3 and by 11% on EdgeBench games under matched budgets. Together, these findings highlight the need for matched-budget baselines and held-out evaluation to distinguish genuine harness improvements from benchmark-specific search and overfitting, and point to a more careful characterization of when automatic harness evolution is actually useful. Our code is available at https://github.com/rethinking-harness-evolution.

Yike Wang, Huaisheng Zhu, Zhengyu Hu +8

Allen Institute for AI · University of Washington · Independent

Large Language Model AgentsTest-Time Scaling
AskPDF
Aug 3, 2026cs.LG

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54% to 66% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.

Luan Zhang, Ruochen Zhou, Dandan Song +9

Beijing Institute of Technology, China · City University of Hong Kong, China · Independent, China

Evolving HarnessAgent HarnessEvolutionary Optimization Methods
AskPDF
Date pendingcs.MA

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?

Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EVOHARNESSBENCH, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EVOHARNESSBENCH places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.

Zixuan Ke, Vaidehi Patil, Haizhou Shi +9

Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison +1

Evolving HarnessAgentic BenchmarksMulti-Llm AgentsForgetting
AskPDF