Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design

Gaotang Li; Bhavana Dalvi Mishra; Yifei Ming; Yanfei Chen; Zhongying CuiZhu; Lesly Miculicich; Long T. Le; Rui Meng

cs.CLOct 20, 2026

Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design

Authors: Gaotang Li, Bhavana Dalvi Mishra, Yifei Ming, Yanfei Chen, Zhongying CuiZhu, Lesly Miculicich, Long T. Le, Rui Meng,

Organizations: University of Illinois Urbana-Champaign · Google Cloud AI Research

Abstract

Agent harnesses are executable code that specifies the prompts, tools, and control flow around a language model. A single global harness can struggle with heterogeneous tasks that require different execution strategies. In this work, we introduce Object-Oriented Harness (OO-Harness), a harness optimization framework that evolves a hierarchy of executable harness classes using object-oriented design. During training, an evolution agent uses execution records to create a family of specialized harnesses, each implemented as a subclass of the base or an existing specialized harness via inheritance. At test time, a harness router uses routing lessons accumulated during training to select a specific harness for each task. Across three long-horizon benchmarks spanning business workflows, workspace tasks, and tool use, OO-Harness achieves the best performance among the compared methods, with an absolute improvement of up to 15% over the base. Extensive ablation studies examine the contributions of its core components.

Figures & tables

Figure 1: Three ways to organize harness adaptation. (a) A single global harness serves all tasks. (b) A router selects among independent harness implementations. (c) OO-Harness selects among specialized harnesses that share code through inheritance.

Figure 2: Problem setting. We use a train/test split: harnesses evolve on discovery tasks and are evaluated on disjoint held-out tasks.

Figure 3: OO-Harness training and test-time execution. (1) Base execution records initialize task clusters. (2) The evolution agent builds and evaluates subclasses, updating lessons from execution feedback. (3) The resulting harness hierarchy, harness descriptions, and routing lessons are frozen for testing. (4) The harness router uses the task, observable base trajectory, and frozen routing lessons to select a specialized harness or reuse the base result.

Test-time scaling
	AutomationBench			Workspace-Bench			Toolathlon-GYM
Method	Mean $\uparrow$	PASS $\uparrow$	Steps $\downarrow$	Mean $\uparrow$	PASS $\uparrow$	Steps $\downarrow$	Mean $\uparrow$	PASS $\uparrow$	Steps $\downarrow$
base	$61.62_{\pm 0.15}$	$25.44_{\pm 0.51}$	$31.27_{\pm 0.13}$	$25.45_{\pm 2.98}$	$3.67_{\pm 0.58}$	$34.41_{\pm 1.34}$	$71.87_{\pm 0.65}$	$31.15_{\pm 0.28}$	$40.50_{\pm 0.99}$
Parallel ( $K{=}2$ ) ‡	$61.46_{\pm 2.38}$	$24.67_{\pm 2.83}$	$31.64_{\pm 0.23}$	$32.54_{\pm 4.30}$	$5.67_{\pm 1.15}$	$41.62_{\pm 2.56}$	$74.85_{\pm 1.45}$	$31.55_{\pm 1.96}$	$39.94_{\pm 0.32}$
Parallel ( $K{=}5$ ) ‡	$60.24_{\pm 1.07}$	$24.00_{\pm 0.47}$	$31.29_{\pm 0.05}$	$37.71_{\pm 0.45}$	$7.67_{\pm 0.58}$	$48.63_{\pm 1.92}$	$75.69_{\pm 0.79}$	$32.14_{\pm 0.56}$	$38.90_{\pm 0.00}$
Sequential ( $K{=}2$ ) ‡	$59.29_{\pm 0.25}$	$25.00_{\pm 1.41}$	$31.86_{\pm 0.12}$	$27.93_{\pm 2.13}$	$3.33_{\pm 2.52}$	$32.91_{\pm 2.07}$	$73.53_{\pm 1.67}$	$33.33_{\pm 2.81}$	$40.97_{\pm 1.32}$

Table 1: Held-out results with a frozen executor. Mean : mean partial credit on AutomationBench, micro-averaged rubric pass rate on Workspace-Bench, and mean reward on Toolathlon-GYM. PASS : strict task success (reward

\geq 0.999

on AutomationBench and Toolathlon-GYM; all rubrics satisfied on Workspace-Bench). The primary metric is Mean for AutomationBench and Workspace-Bench, and PASS for Toolathlon-GYM. Steps counts only the finally selected trajectory, excluding other rollouts and auxiliary selection calls. Values are mean

\pm

standard deviation over three runs; scores are percentages. Bold marks each column’s maximum score (

\uparrow

) or minimum steps (

\downarrow

). † : supervised harness evolution with discovery-task feedback. ‡ : unsupervised test-time scaling of the base harness. Held-out grader feedback is unavailable to all methods.

Figure 4: Global versus hierarchical harness optimization. Compared with Meta-Harness , OO-Harness shows (a) larger early training gains, (b) shorter recorded search time, and (c) higher held-out scores. Curves align runs by their fraction of recorded updates.

		OO-Harness routing evidence
Benchmark / metric	Base	Task only	Task + trajectory
Automation / Mean	$61.30$	$64.88$	$\mathbf{77.52}$
Workspace / Mean	$24.57$	${40.03}$	$\mathbf{40.56}$
Toolathlon / PASS	$31.35$	$33.33$	$\mathbf{37.70}$

Table 2: Routing ablation on a frozen OO-Harness catalogue. Task only uses the task description and available task metadata; Task + trajectory also observes the base execution. Base executes without routing. Entries report primary scores (%) from single ablation runs, distinct from the repeated runs in Table 1 .

Figure 5: Inheritance ablation on Workspace-Bench. We find that learning through inheritance outperforms Flat and Base-reset.

		Original Base performance
Routing destination	Tasks	Mean reward	Strict PASS
Reuse Base	134	77.15	36.57
Specialized harness	166	44.40	14.25

Table 3: Base performance by routing destination on AutomationBench. We find that tasks routed to specialized harnesses have substantially weaker Base performance on average, while OO-Harness preserves stronger Base executions through reuse.

Figure 6: Learned clusters and an example router lesson. Panels (a,b) compare learned clusters (rows) with domains or task types (columns); cells count shared discovery tasks. AutomationBench clusters largely reflect execution needs, while Workspace-Bench clusters largely follow task types. Panel (c) schematically shows how repeated query-parsing errors in a Base trajectory match a saved routing lesson, leading to selection of an API error-normalizer harness. Blank cells are zero; Workspace-Bench column IDs match the numbered rows.

Figure 7: Base reuse on the selected OOD test tasks. After evolution on the disjoint training split, OO-Harness reuses Base on every test task, preserving its performance, while applying the learned Meta-Harness harness reduces PASS.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Generating a Word document through the file backend. For the same illustrative write_file call, the base backend saves text under a .docx filename; the specialized backend renders the text into a binary document.

Figure 9: Adding execution checks through inheritance. Successive nodes add a missing-write completion check, a near-budget reminder, and tool-error guidance. Arrows show inheritance. The filled node s29 is active; s9 and s12 are inactive ancestors that supply inherited behavior.

Base execution	Routing lesson	Router choice
A lead-scoring execution makes no writes. HubSpot and Intercom return 401 errors indicating no connected account; a Lead lookup returns 404. The final response is empty.	Repeated connection errors and failed resource lookups suggest that information about connected services and available resources may help.	Resource-manifest harness.
Another execution completes three writes, has no errors, and finishes cleanly.	Successful writes and a clean finish support keeping the Base result.	Reuse Base.

Appendix

Table 4: Execution evidence behind two routing choices. Examples from an earlier AutomationBench workflow pair observed Base behavior with the applicable routing lesson and the recorded choice.

Figure 10: Learned task clusters on Toolathlon-GYM. Rows are learned clusters and columns are evaluated AAH app groups. Cells count shared discovery tasks, showing how a cluster can span several applications. Blank cells are zero; Base and unclustered tasks are omitted.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 183 papers

Aug 3, 2026

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54% to 66% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.

Luan Zhang, Ruochen Zhou, Dandan Song +9

Beijing Institute of Technology, China · City University of Hong Kong, China · Independent, China

Agent Harness Optimization LLM Agents LLM Agent Harnesses

Ask PDF

Sep 1, 2026

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16

LLM Agent Evaluation Agent Harness Optimization AI Agent Benchmarks

Ask PDF

Jun 12, 2026

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.

Tingyang Chen, Shuo Lu, Kang Zhao +11

Darwin Agent Team

LLM Agent Orchestration Agent Harness Optimization LLM Agent Workflow Optimization LLM Agent Self-Improvement LLM Agent Harnesses Self-Evolving Agents

Ask PDF

Abstract

Agent Harness Optimization

Explore similar work

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry