Harmony Harness: Multi-Objective Self-Evolution of Agent Harnesses

Weichen Yu; Lesly Miculicich; Bhavana Dalvi Mishra; Jiefeng Chen; Shuo Li; Matt Fredrikson; Tomas Pfister; Long T. Le

cs.AIOct 10, 2026

Harmony Harness: Multi-Objective Self-Evolution of Agent Harnesses

Authors: Weichen Yu, Lesly Miculicich, Bhavana Dalvi Mishra, Jiefeng Chen, Shuo Li, Matt Fredrikson, Tomas Pfister, Long T. Le

Organizations: Google Cloud AI Research · Carnegie Mellon University · Google DeepMind

Abstract

Recent work on self-evolving LLM agents treats the agent harness as an optimization target that can be iteratively improved through execution feedback. Existing approaches, however, largely optimize a single objective and evolve the harness as a monolithic artifact, limiting their ability to handle competing goals and to separate transferable design from environment-specific adaptation. We introduce Harmony Harness, a framework for multi-objective, compositional harness evolution. Harmony Harness formulates harness evolution as a multi-objective optimization problem and uses decomposition-based evolutionary search to maintain solutions across different regions of the Pareto frontier, avoiding reliance on a hand-designed structural diversity metric for repository-scale harnesses. In parallel, we introduce factorized evolution operators that separately refine general-purpose harness mechanisms and benchmark-specific adaptations, together with a dynamic composition workflow that integrates them at inference time. Across SecRepoBench and BaxBench, Harmony Harness consistently outperforms strong harness-evolution baselines, with gains exceeding 10% on both benchmarks. Our results show that explicitly modeling both objective trade-offs and harness structure yields substantially stronger and more transferable agent evolution than existing single-objective, monolithic approaches.

Figures & tables

Figure 1: Overview of Harmony Harness . (a) The functionality–security space is split into weighted subproblems. Each subproblem keeps a harness and together they approximate the Pareto frontier. (b) An LLM proposer diagnoses failures and forms a hypothesis. Operators edit tools, skills, or logic, and place each edit in either the general scope or the environment-specific scope. The child is validated before evaluation. (c) At inference, a composition agent picks components from the general and environment-specific pools to build a task-specific harness.

Panel of Figure 2 — Figure 2: Evolution results per split on (a) SecRepoBench and (b) BaxBench.

	SecRepoBench			BaxBench
Method	$\Delta_{\text{evolve}}$	$\Delta_{\text{OUT}}$	retained	$\Delta_{\text{evolve}}$	$\Delta_{\text{OUT}}$	retained
Meta-harness (func-only)	$+3.3$	$-3.4$	$-103\%$	$+3.6$	$-7.1$	$-197\%$
Meta-harness (sec-only)	$+13.3$	$+3.3$	$25\%$	$+17.9$	$+3.6$	$20\%$
Meta-harness (func & sec)	$+16.6$	$+3.3$	$20\%$	$+14.3$	$+3.6$	$25\%$
DGM (func & sec, scalar)	$+20.0$	$+0.0$	${0\%}$	$+10.7$	$+3.6$	$34\%$
Harmony Harness (func & sec)	$+36.6$	$+20.0$	$\mathbf{55\%}$	$+21.4$	$+10.7$	$\mathbf{50\%}$

Table 1: How much evolved security survives a distribution shift.

\Delta_{\text{evolve}}

is a method’s security gain over the un-evolved baseline on the split it evolved on;

\Delta_{\text{OUT}}

is its gain over the same baseline on held-out-OUT; retained

=\Delta_{\text{OUT}}/\Delta_{\text{evolve}}

. Mean over 3 runs.

Strategy	Acceptance rule	Func	Sec	F&S
Static (no evolution)	—	66.7	56.7	46.7
Functionality-only	accept if $F\!\uparrow$	73.3	60.0	56.7
Safety-only	accept if $\text{viol.}\!\downarrow$	70.0	66.7	46.7
Sequential (func $\to$ sec, sec $\to$ func)	one objective per generation, then the other	70.0	76.7	70.0
Prompt-merge variants	merge func & safety prompts	66.7	60.0	50.0
Joint-weighted ( $\alpha\!\in\!\{.25,.5,.75\}$ )	$\alpha F\!+\!(1\!-\!\alpha)S$	70.0	73.3	70.0

Table 2: Evolution strategies on SecRepoBench. Func/Sec/F&S in %. Mean over 3 runs.

Figure 3: Cross-benchmark transfer. Each cell is one (optimized-on $\times$ evaluated-on) pair, split into meta-harness (left) and Harmony Harness (right); dashed outlines mark in-domain cells, solid outlines zero-shot transfer.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Problem formulation (§ 3 )
$\mathcal{H}$ , $H$	Space of agent harnesses; a harness
$\mathcal{D}=\{t_{n}\}_{n=1}^{M}$	Evaluation task set of $M$ tasks
$t_{n}=(x_{n},p_{n})$	Task $n$ : task specification $x_{n}$ and security policy $p_{n}$
$\tau\sim P_{H}(\tau\mid x_{n})$	Trajectory of harness $H$ on task $t_{n}$ (conditioned on the visible spec $x_{n}$ )
$F(\tau,x_{n})\in\{0,1\}$	Functional success: the solution passes the tests of $x_{n}$

Appendix

Table 3: Summary of notation.

Operator	Writable directories	Required deliverable
tool	tools/ , utils/ (plus config/ )	A change to how the agent obtains or consumes ground truth , covering one of four cases: no real tool is wired into the loop; a tool is called but its result is wasted (the log is ignored, or only the build is checked and never the server’s startup); the tool itself returns too little to act on; or a helper in utils/ discards the critical information before it reaches the agent. Phantom verification is explicitly forbidden.
skill	skills/ , prompts/	One reusable skill: a trigger condition, a short general procedure, and one worked example. The skill is a single .py module dropped into the scope directory, with no registration needed. Skills written as a default behavior with an escape clause (e.g. “unless the spec says otherwise, return 200”) are forbidden. In one of our runs, such a skill fixed its target task but broke two unrelated ones.
logic	steps/ , orchestrator/ ; at general scope, also the loop modules at the package root	A change to the control flow : add a missing stage, act on a branch that is currently ignored, adjust round counts and budgets (the values go in config/ ), or change the DAG itself. Achieving the same effect by editing prompts is not allowed.

Appendix

Table 4: The three scoped variation operators. Each operator may write only its own directories, at the scope chosen before the edit; any write outside them is reverted.

Strategy	Acceptance rule	Func	Sec	F&S
Joint Evolving	Tchebycheff over $(F,S)$	70.0	76.7	63.3
+ factored evolving	+ factorized operators (§ 4.2 )	70.0	83.3	66.7
+ dynamic orchestration	+ dynamic composition (§ 4.3 )	76.7	93.3	76.7

Appendix

Table 5: Component ablation of joint evolution on SecRepoBench (search split, %). Each row adds one component to the row above. The bottom row (+ dynamic orchestration) is the full method and is repeated from Table 9 (mean over 3 runs).

Orch.	Operator pool	Seed	Gen-3 children	Best
✓	tool , skill , logic	66.7	83.3, 83.3, 33.3	83.3
✓	+ generalize	66.7	100.0, 50.0, $\dagger$	100.0
✗	tool , skill , logic	33.3	66.7, 66.7, 66.7	66.7
✗	+ generalize	33.3	66.7, 83.3, 66.7	83.3

Appendix

Table 6: Operator-pool ablation (F&S, %). “Gen-3 children” lists the F&S of each child produced in the third generation; “Best” is the best F&S reached by the run.

\dagger

: run that leads to fallback.

Figure 5: Search-split Pareto front of the MOEA/D-alone run (Table 13 ): the baseline and all

15

children, shaded by generation. After generations 1–2 the front holds two non-dominated harnesses (dashed); in generation 3, moead_g003_s01 reaches the ideal point

z^{\ast}

and dominates the rest (solid). Children with identical scores are offset horizontally so that each stays visible. An interactive version is available at https://anonymous.4open.science/w/HH_visualization-4F69/index.html .

Figure 6: Search-set Pareto front of a MOEA/D run on BaxBench (

N=6

subproblems, 28-task cover set, generations 1–4): the two seed harnesses and all

24

children, shaded by generation. After generation 1 the front is a single harness (dashed); from generation 2 on it holds two mutually non-dominated harnesses (solid). Children with identical scores are offset horizontally so that each stays visible. An interactive version is available at https://anonymous.4open.science/w/HH_visualization-4F69/baxbench.html .

Evolution surface	Held-out-IN	Held-out-OUT	comb.
Monolithic summarize $\to$ rewrite	40.0	30.0	35.0
Factored: evolving tools only	46.7	33.3	40.0
Factored: skills/logic only	50.0	46.7	48.3
Factored: tools $+$ skills/logic	60.0	56.7	58.3
$+$ dynamic orchestration (full)	86.7	66.7	76.7

Appendix

Table 7: Evolution-surface ablation (held-out combined F&S, %). Same objective, budget, and seed; only what is evolved changes. “comb.” is the unweighted mean of the two 30-task splits. The full row is repeated from Table 9 (mean over 3 runs).

Execution	Nodes	Sec	Func	F&S	Avg. nodes	Avg. tokens
Baseline (no evolution)	—	50.0	60.7	42.9	—	31,902
Fixed linear pipeline (orchestrator off)	5	53.6	71.4	53.6	5.0	58,230
Dynamic workflow (orchestrator on)	5–8	64.3	75.0	64.3	6.2	63,064

Appendix

Table 8: Dynamic orchestration ablation on BaxBench ( USE_ORCHESTRATOR on/off). The lower two rows are the same evolved harness, seed, and budget, differing only in the toggle; the un-evolved baseline is repeated from Table 10 for reference. “Nodes” is the range of plan sizes observed over the 28 tasks (Figure 8 ; the planner may emit up to 16), “avg. nodes” the mean number actually executed, and “avg. tokens” the mean solve-time token spend per task.

Method	Search			Held-out In			Held-out Out			Avg. F&S
Method	Sec	Func	F&S	Sec	Func	F&S	Sec	Func	F&S	Avg. F&S
Baseline	56.7	66.7	46.7	66.7	73.3	50.0	46.7	70.0	36.7	44.4
Meta-harness (func & sec)	73.3	70.0	53.3	70.0	73.3	56.7	50.0	70.0	43.3	51.1
Meta-harness (func-only)	60.0	73.3	56.7	60.0	73.3	50.0	43.3	70.0	36.7	47.8
Meta-harness (sec-only)	70.0	66.7	46.7	66.7	60.0	53.3	50.0	63.3	33.3	44.4
DGM (func & sec, scalar)	76.7	73.3	60.0	66.7	70.0	53.3	46.7	66.7	33.3	48.9

Appendix

Table 9: Evolution on SecRepoBench (%). Best result per column in bold . Mean over 3 runs.

	Search			Held-out IN			Held-out OUT			Avg.
Method	Sec	Func	F&S	Sec	Func	F&S	Sec	Func	F&S	F&S
Baseline (no evolution)	50.0	60.7	42.9	50.0	62.4	42.9	46.4	57.1	42.9	42.9
Meta-harness (func & sec)	64.3	71.4	60.7	57.1	62.4	53.6	50.0	60.7	42.9	52.4
Meta-harness (func-only)	53.6	71.4	46.4	53.6	64.3	50.0	39.3	60.7	39.3	45.2
Meta-harness (sec-only)	67.9	50.0	46.4	60.7	57.1	53.6	50.0	53.6	42.9	47.6
DGM (func & sec, scalar)	60.7	71.4	57.1	50.0	64.3	50.0	50.0	57.1	39.3	48.8

Appendix

Table 10: Evolution on BaxBench (%, secure web-app generation). Same protocol as Table 9 . “Average” is the mean F&S over the three splits. Mean over 3 runs.

Language	$n$	One-shot (cc)			$+$ HH harness (ours)
Language	$n$	Sec	Func	F&S	Sec	Func	F&S
C	93	75.3	88.2	75.3	89.2	90.3	89.2
C++	63	76.2	98.4	76.2	90.5	93.7	90.5
Go	57	70.2	87.7	70.2	89.5	89.5	89.5
JavaScript	69	72.5	89.9	72.5	87.0	87.0	82.6
Python	75	74.7	96.0	70.7	90.7	89.3	85.3

Appendix

Table 11: HH evolved on BaxBench , transferred zero-shot to CWEval (%), against the un-evolved one-shot model with Claude Code harness (cc) on the same tasks. Better of the two harnesses per metric and per language in bold . Mean over 3 runs.

Method	Evolved on	Evaluated on	Sec	Func	F&S
Baseline (no evolution)	—	BaxBench	50.0	60.7	42.9
Meta-harness	SecRepoBench	BaxBench	53.6	57.1	39.3
HH (ours)	SecRepoBench	BaxBench	59.6	64.3	57.1
Baseline (no evolution)	—	SecRepoBench	56.7	66.7	46.7
Meta-harness	BaxBench	SecRepoBench	43.3	50.0	36.7
HH (ours)	BaxBench	SecRepoBench	80.0	70.0	70.0

Appendix

Table 12: Cross-benchmark transfer (%). A harness is evolved on one benchmark and evaluated on the other with no further evolution, on the target’s search split (the split whose baseline appears in Tables 9 and 10 ); the baseline row is the un-evolved harness on that split. Mean over 3 runs.

stage	hypervolume	$z^{\ast}$ (func, sec)	most-balanced harness
init (baseline)	0.321	(56.7, 56.7)	baseline (56.7, 56.7)
generation 1	0.528	(70.0, 76.7)	moead_g001_s00 (70.0, 70.0)
generation 2	0.528	(70.0, 76.7)	moead_g001_s00 (70.0, 70.0)
generation 3	0.537	(70.0, 76.7)	moead_g003_s01 (70.0, 76.7)

Appendix

Table 13: Per-generation frontier quality for MOEA/D alone (

N=5

T=2

, 3 generations; no factored evolving or dynamic orchestration, to isolate the decomposition) on the SecRepoBench search set. Hypervolume is the area of the population’s Pareto front against reference point

(0,0)

;

z^{\ast}

is the ideal point (per-objective running best); “most-balanced” is the population member with the highest worst objective (

\max\min(\text{func},\text{sec})

	Held-out IN (%)			Held-out OUT (%)			comb.
harness	Func	Sec	F&S	Func	Sec	F&S	F&S
baseline	73.3	66.7	50.0	70.0	46.7	36.7	43.3
without env-feedback	—	—	40.0	—	—	30.0	35.0
feedback-env gen1 best	63.3	80.0	56.7	66.7	56.7	43.3	50.0
feedback-env gen2 best	70.0	80.0	60.0	80.0	63.3	56.7	58.3
feedback-env gen3 best	80.0	83.3	66.7	76.7	50.0	46.7	56.7

Appendix

Table 14: Held-out generalization (30 tasks each, eval-only). IN

=

crash-type families seen during search; OUT

=

unseen families. The feedback-tool per-generation bests beat baseline on combined F&S (

58.3

43.3

); the search-seeded winner is worse than baseline (

35.0

Figure 7: Population diversity over

137

evolved BaxBench harnesses (

34

run tags). Left: the five components and their equal-weight composite (

\mathrm{DIVERSITY}=0.300

, dashed). The index is held up by mean pairwise distance

D_{\text{struct}}

and label entropy

H_{\text{label}}

and held down by nearest-neighbour distance

D_{\text{novel}}

and the fraction of distinct solutions

N_{\text{eff}}/n

. Middle:

N_{\text{eff}}

against the duplicate threshold

\epsilon_{\mathrm{dup}}

;

N_{\text{eff}}

falls steeply (

112

classes at

\epsilon_{\mathrm{dup}}=0.001

down to

9

\epsilon_{\mathrm{dup}}=0.05

) and flattens only briefly (still

9

0.1

), so “how many distinct solutions” has no threshold-free answer and the reported

9

describes the scale of a typical edit. Right: the distance distributions behind the headline—over all

\binom{137}{2}

pairs the mass sits far from zero (top), while over each genome’s nearest neighbour it concentrates in the first bin (bottom, note the change of vertical scale).

Quantity	Value	Reading
$d_{\text{tok}}$ / $d_{\text{ast}}$ / $d_{\text{lin}}$	$0.417$ / $0.162$ / $0.433$	AST distance is $2.6\times$ smaller than token and line distance: edits move text and individual statements, rarely control structure
$D_{\text{struct}}$	$0.345\pm 0.269$	globally the population is well spread
$D_{\text{novel}}$	$0.014$	… but nearest neighbours are close in code
$N_{\text{eff}}$ / $n$	$9/137=6.6\%$	nine equivalence classes at $\epsilon_{\mathrm{dup}}=0.05$
$N_{\text{eff}}$ sweep	$0.001{\to}112$ $0.005{\to}68$ $0.01{\to}30$ $0.05{\to}9$ $0.1{\to}9$ $0.2{\to}7$
$H_{\text{label}}$	$0.934$	134/137 labelled, 17/17 labels hit, $\mathrm{Simpson}=0.914$

Appendix

Table 15: Pooled population,

n=137

variants from 34 run tags,

g_{\mathrm{sh}}=5

, weights

(0.4,0.3,0.3)

\epsilon_{\mathrm{dup}}=0.05

Label	Frac.	Label	Frac.	Label	Frac.
build-verify	$0.78$	skill	$0.53$	parsing	$0.30$
planning	$0.70$	config-knob	$0.45$	self-critique	$0.25$
security	$0.64$	tool-add	$0.41$	error-handling	$0.14$
loop-control	$0.62$	test-gen	$0.40$	context-mgmt	$0.12$
prompt-edit	$0.61$	retry	$0.36$	edit-format	$0.08$
		input-validation	$0.30$	model-params	$0.01$

Appendix

Table 16: Mechanism-label distribution over the 134 labelled children. Labels are not exclusive; the fraction is the share of children carrying that label.

Flag	Default	Effect
--ngram	$5$	token shingle length $g_{\mathrm{sh}}$
--weights	0.4,0.3,0.3	token, AST, line weights; renormalized
--dup-threshold	$0.05$	$\epsilon_{\mathrm{dup}}$ for $N_{\text{eff}}$ and the near-duplicate list
--include-seeds	off	add bax_repo_seed / bax_wf_seed
--top	$5$	length of the most-novel and closest-pair lists
--json	—	write every metric, including the per-genome nearest-neighbour distances, the near-duplicate pairs and the per-tag breakdown

Appendix

Table 17: Options.

ID	Mechanism fitted	Injected edit
Tier 1 — lexical
P1	task identity	if task_id in {...} guarding a bespoke code path, with evolve-split IDs enumerated
P2	crash-type string	a crash-type name drawn from the evolve split hardcoded into the prompt template
P3	repository name	if "libpng" in repo_path selecting a hand-written fix recipe
P4	task text	a skill file whose body names specific evolve-split tasks
Tier 2 — structural

Appendix

Table 18: The planted-overfit taxonomy. Each plant is injected into a real admitted child as the sole modification. Tier 1–2 labels are certain by construction; Tier 3 and class S labels are verified behaviourally (§ K.4 ).

Figure 8: Workflow DAGs planned by the evolved orchestrator over

28

BaxBench tasks. Solid edges are unconditional; dashed nodes/edges fire only under a when guard. Plans range from

5

8

nodes and are re-planned per task, rather than fixed at harness-design time.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 185 papers

Aug 3, 2026

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54% to 66% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.

Luan Zhang, Ruochen Zhou, Dandan Song +9

Beijing Institute of Technology, China · City University of Hong Kong, China · Independent, China

Agent Harness Optimization LLM Agents LLM Agent Harnesses

Ask PDF

Sep 29, 2026

Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer

A harness is the code around a language-model agent that organizes prompts, calls tools, manages context, and controls execution. As models grow stronger, recent work has begun to let agents improve their own harnesses, a line of work known as self-evolving harnesses. In most existing methods, a separate proposer running on a human-designed harness modifies the solver's harness, and a separate harness is evolved for each benchmark. Real-world tasks come from many domains, so both the evolution and the evaluation of a harness should cover a diverse range of tasks. We propose a framework close to recursive self-improvement: the same frozen model, on the same version of the harness, first solves tasks as the solver and then, as the proposer, reads the complete run records and directly edits the harness that runs it. Each evolution batch draws tasks from five benchmarks in different domains. To measure generalization, training and held-out tasks are strictly separated, and we additionally evaluate on five out-of-distribution benchmarks never used during evolution. We frame the evolution process as deep-learning training with two stages, multi-task pretraining and continual training. Starting from a 49-line seed harness, the harness obtained at the end of the first stage improves the average score by 4.48 points on the in-distribution benchmarks and by 12.64 points on the out-of-distribution benchmarks, surpassing Codex on the former and matching it on the latter. In the second stage, continued evolution on Claw-Eval, one of the out-of-distribution benchmarks, further raises the score on that benchmark from 66.17 to 68.06, exceeding Codex. We also provide an in-depth analysis of the mechanisms that emerged during evolution, including output truncation, history compaction, and independent review.

Qiankai Xu

Nanjing University

Continual Learning for LLM Agents AI Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization LLM Agent Self-Improvement

Ask PDF

Sep 3, 2026

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?

Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EvoHarnessBench, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EvoHarnessBench places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.

Zixuan Ke, Vaidehi Patil, Haizhou Shi +9

Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison +1

Continual Learning for LLM Agents LLM Agent Harnesses Self-Evolving Agents Agent Benchmarks

Ask PDF

Abstract

Agent Harness Optimization

Explore similar work

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer

EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?