LiteEvo: Automated, Cost-Efficient Harness Evolution for Generalization to Unseen Tasks

Euntae Choi; Sumin Song; Sungjoo Yoo

cs.AISep 27, 2026

LiteEvo: Automated, Cost-Efficient Harness Evolution for Generalization to Unseen Tasks

Authors: Euntae Choi, Sumin Song, Sungjoo Yoo

Organizations: Seoul National University

Abstract

An LLM agent is defined by two things: the weights inside its model and the harness of components assembled around it. Harnesses are still handcrafted, and HarnessX, which evolves them automatically, starts each benchmark from a handcrafted harness, reports gains on the tasks it evolved on, and budgets 100 to 175 million meta-agent tokens per benchmark. We propose LiteEvo, a lightweight harness-evolution algorithm whose tool-free meta-agents mine agent trajectories for reusable components, curate them into a versioned library, and compose each round's harness from it, starting every benchmark from the same neutral harness and never naming the benchmark. Evolving on the graded tasks of five agentic benchmarks with a frozen Qwen3.5-9B, LiteEvo lifts pass@2 by 10.5 to 67.7pp and reaches comparable or higher pass@2 than a reproduction of HarnessX (71.0 against 67.3 on average) at 13.0 lower mean API cost. Harnesses evolved on train tasks keep their gains on unseen test tasks of four benchmarks, and LiteEvo also lifts Claude Code with Sonnet 4.6 by 1.2 to 71.4pp.

Figures & tables

Panel of Figure 1 — Figure 1: LiteEvo reaches comparable or higher pass@2 than our reproduction of HarnessX at 13.0 $\times$ lower mean API cost and lifts a frozen 9B agent to the level of frontier agents on ALFWorld and AppWorld. (a) Test pass@2 against the API cost of one evolution run, both methods transductive. Error bars span the three test runs, and the black markers average the five benchmarks. (b) Qwen3.5-9B given only the task text, and with the harness LiteEvo evolves (inductive, except on GAIA), against Claude Code with Sonnet 4.6 at low and high effort and GPT-5.4 with the static, unevolved harness that HarnessX reports ( Chen et al., 2026 ) , on HarnessX’s own task sample for WebShop. Solid bars give pass@1 and hatched bars pass@2. †: Sonnet 4.6 is told how to finish a task; the agent under LiteEvo never is.

Figure 2: Overview of LiteEvo. (0) Every benchmark starts from the same neutral harness $h_{0}$ . (1) Frozen workers run the current harness $h_{t}$ on the train tasks, and failure-oriented group sampling keeps ten mixed-outcome and three all-fail task groups as evidence. (2) Four meta-agents mine failure modes, curate a versioned component library († marks the strategies, which advise the composer and never reach the agent) and compose a candidate. (3) A pass@2 gate on the val tasks accepts, holds or reverts the candidate. (4) The frozen agent runs the selected harness $h_{t+1}$ in the next round.

Benchmark	Mode	Method	Initialization	pass@2 (%)		$\Delta$	Best Round	API cost ($)
				Initial	Evolved
ALFWorld	Bare	–	Task only	37.1		–	–	–
	Transductive	HarnessX	+Seed	64.2	92.3	+28.1	6	118.47
	Transductive	LiteEvo	+Neutral	60.9	96.3	+35.3	5	8.24
	Inductive	LiteEvo	+Neutral	60.9	94.8	+33.8	4	9.17
WebShop	Bare	–	Task only	24.0		–	–	–

Table 1: Test pass@2 (%) of Qwen3.5-9B, averaged over three test runs. Bare gives the agent only the task text. Initial is the starting harness, a benchmark-specific seed for HarnessX (+Seed) and the neutral harness for LiteEvo (+Neutral), and Evolved is the returned harness

h_{t}

, first run in round

t

( Best Round ).

\Delta

is Evolved minus Initial in pp, computed before rounding.

	ALFWorld		WebShop		AppWorld		GAIA		SWE-bench Verified
Harness	P@1	P@2	P@1	P@2	P@1	P@2	P@1	P@2	P@1	P@2
Bare	25.1	37.1	14.5	24.0	0.0	0.0	0.0	0.0	9.1	16.4
Bare + format	46.8	56.7	9.3	13.8	0.0	0.0	29.1	44.3	50.9	61.2
+Processor	49.3	57.7	41.3	53.0	0.0	0.0	44.6	59.2	51.0	63.1
+Prompt	92.8	95.5	51.0	59.0	46.1	63.1	41.9	49.5	49.4	58.2
Full	92.6	96.3	54.0	62.7	51.9	67.7	53.6	65.2	54.2	63.2

Table 2: The returned harness ( Full ) split into its system prompt and processors (Qwen3.5-9B, transductive): +Prompt and +Processor add one part to Bare + format , the bare agent with a minimal action-format hint. P@1 and P@2 are pass@1 and pass@2 in %, averaged over three test runs (one for Bare on WebShop, GAIA and SWE-bench Verified).

Benchmark	Mode	Method	pass@2 (%)		$\Delta$	Best Round	API cost ($)
			Initial	Evolved
ALFWorld	Transductive	LiteEvo	88.8	96.3	+7.5	2	8.21
ALFWorld	Inductive	LiteEvo	88.8	95.5	+6.7	7	12.92
WebShop	Transductive	LiteEvo	77.0	81.0	+4.0	6	9.84
WebShop	Inductive	LiteEvo	77.0	81.0	+4.0	5	12.57
AppWorld	Transductive	LiteEvo	0.0	66.1	+66.1	7	11.09

Table 3: pass@2 (%) of Claude Code with Sonnet 4.6 at low effort and its own tools turned off, given only the task ( Initial ) and with the harness that LiteEvo evolves for it; each evolved cell is a single measurement. The GAIA run blocks known copies of the benchmark’s answers. In Figure 1 b, Sonnet 4.6 on AppWorld is also told how to finish a task.

Round		0	1	2	3	4	5	6	7	Total
Librarian	Add	19	11	12	10	6	11	8	8	85
	Update	0	4	4	2	5	0	3	7	25
	Merge	0	1	3	2	0	1	0	0	7
	Drop	0	0	1	0	0	0	0	1	2
Library	Removed	0	2	4	2	0	0	0	1	9
Library	Size	3.8	5.8	8.0	10.0	11.2	13.4	15.0	16.4	–

Table 4: Library operations and composer selections per round in the five transductive LiteEvo runs (Qwen3.5-9B), summed over the runs; Size (components selectable after the round) and Picked (selected for the next harness) are averaged. Removed counts components deleted by merges and drops, and Pick rate is the pooled ratio of Picked to Size.

	AppWorld		GAIA
Initial harness	P@1	P@2	P@1	P@2
Neutral (default)	51.9	67.7	53.6	65.2
HarnessX seed	49.1	64.9	48.4	59.9

Table 5: Neutral or handcrafted start (Qwen3.5-9B, transductive, one run each; HarnessX’s own seed on GAIA): test P@1 and P@2 (%), mean of three test runs.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Starting harness	What the optimizer is told about the benchmark	Evaluation	Cost reported
HarnessX	handcrafted for each benchmark	its benchmark-specific seeds	L0	token budget
Meta-Harness	per-domain seeds	a skill written by hand for each domain	L0 on its agentic benchmark; held-out and new datasets for classification	iterations, hours
AutoHarness	one template	the game’s name and description	new random instances of each game	iterations
AHE	a single shell tool	an explore agent aimed at the benchmark	L0, plus one other benchmark without re-evolution	hours
HarnessCompass	a single shell tool	software-repair hints in the meta prompt	L1 on one benchmark, plus a second model	iterations
ACE	the benchmark’s official agent	a reflector named after the benchmark	L1 offline; online on the test stream	latency, rollouts

Appendix

Table 6: How LiteEvo and the closest methods start, what their optimizer is told about the benchmark, how far from the evolution tasks they are evaluated, and what optimizer cost they report. Levels: L0, evolved and graded on the same tasks; L1, unseen tasks of the same benchmark; L1.5, tasks that need unseen apps; L2, a sibling benchmark; L3, a foreign benchmark. Evo-Bench is a benchmark of evolver models rather than a method.

Figure 3: The neutral harness

h_{0}

, the system prompt with which every run starts. Line breaks inside paragraphs are added for display.

Knob	Neutral	Effect
Protocol	tagged	tagged text replies, or one generic act tool call
Available actions	shown	append the environment’s list of legal actions to each observation
Observation cap	off	truncate long observations (command actions only)
Action fallback	on	without a tag, take the reply’s last line as the action (command actions only)
Remaining turns	hidden	prefix each observation with the number of turns left

Appendix

Table 7: The five runner knobs, their values in the neutral harness, and their effect.

Kind	Body	Placed	Acts
Prompt fragment	an instruction or a worked example	system prompt, after the frame, in its order	every model call
Knowledge	facts observed in trajectories	system prompt, after the fragments	every model call
Processor	a Python function advise(ctx)	one runner hook	each turn it returns a key
Knob	runner settings	read by the composer only	never directly
Strategy	advice on composing	the composer’s and librarian’s input	never reaches the agent

Appendix

Table 8: Component kinds, where the assembly places each, and when it acts.

Meta-agent (model)	Reads	Returns
Analyst (Sonnet 5)	the running harness’s settings and full system prompt; outcome statistics over the round; 13 task groups with all $M$ trajectories each	a task signature; failure modes with their share, evidence and mechanism; candidate components, with a cited step for each fact
Editor (Opus 4.6)	the analyst’s output with its citation checks; a journal of earlier rounds (scores, decisions, library edits, rejected processors)	the selected candidates, rewritten or merged; dropped candidates with reasons; then the processor check
Librarian (Opus 4.6)	the library index with usage statistics and all bodies; the editor’s selections and drops	operations: add, update or merge a complete component, or drop one, each with a reason
Composer (Opus 4.6)	the task signature; this round’s score and the best harness’s; the strategies; the library; the journal; the best harness’s selection, knobs and size (not its frame); the editor’s picks; a 4,000-character excerpt of failed trajectories	a selection of components, a frame and five knob values

Appendix

Table 9: What each meta-agent reads and returns in one round, in call order. Instructions are omitted.

Figure 4: The part of the instructions that all four meta-agents share (1 of 4).

Figure 5: The part of the instructions that all four meta-agents share (2 of 4).

Figure 6: The part of the instructions that all four meta-agents share (3 of 4).

Figure 7: The part of the instructions that all four meta-agents share (4 of 4).

Figure 8: The analyst’s instructions (1 of 2).

Figure 9: The analyst’s instructions (2 of 2).

Figure 10: The editor’s instructions.

Figure 11: The librarian’s instructions.

Figure 12: The composer’s instructions.

Setting	Value
Agent	Qwen3.5-9B in thinking mode; temperature 1.0, top- $p$ 0.95, top- $k$ 20, presence penalty 1.5
Generation and context	4,096 tokens per reply; context window 40,960 tokens; prompt budget 32,768 tokens
Turn caps	20, 20, 40, 20 and 200 (ALFWorld, WebShop, AppWorld, GAIA, SWE-bench Verified)
Rounds and workers	$R=8$ (round 0 and seven evolution rounds), $M=4$
Val rollouts	4 per task (inductive); the train rollouts (transductive)
Analyst sample	10 mixed and 3 all-fail task groups, each with all $M$ trajectories

Appendix

Table 10: Settings of the LiteEvo runs. Exceptions: SWE-bench Verified uses

M=2

, five mixed plus two all-fail groups and two val rollouts per task.

Benchmark	Turn cap	Train	Val	Test
ALFWorld	20	134 (train)	140 (valid-seen)	134 (valid-unseen)
WebShop	20	100	100	100
AppWorld	40	90 (train)	57 (dev)	168 (test-normal)
GAIA	20	–	–	103 (transductive only)
SWE-bench Verified	200	55	55	55

Appendix

Table 11: Task sets. Inductive runs evolve on the train tasks, gate on the val tasks and grade on the test tasks. Transductive runs use the test tasks for all three.

Benchmark	Seed harness	Evolution recipe	Meta-agent reasoning	Other deviations
ALFWorld	ours	ours	thinking off	task seeds cover 88 of the 134 test games
WebShop	ours	ours	thinking off	BM25 search
AppWorld	ours	ours	thinking off	not in the HarnessX paper; no early stopping
GAIA	official	official, patched (pass@2)	official (extended thinking)	a different search API; answer judge off
SWE-bench Verified	official	ours	thinking off	200 turns (release: 60); our grader

Appendix

Table 12: Provenance of each part of our HarnessX runs. Official is the released code, unmodified; patched adds our changes; ours is absent from the release and written by us. The meta-agent is Claude Opus 4.6 on every benchmark.

Benchmark	Mode	pass@2 (%)		$\Delta$	Best Round	API cost ($)
		Initial	Evolved
ALFWorld	Transductive	44.8	88.3	+43.5	7	7.78
ALFWorld	Inductive	44.8	87.3	+42.5	7	7.86
WebShop	Transductive	15.2	65.0	+49.8	4	9.06
WebShop	Inductive	15.2	62.3	+47.2	2	11.39
AppWorld	Transductive	0.0	33.9	+33.9	6	11.32

Appendix

Table 13: Test pass@2 (%) of Qwen3.5-4B with the harness that LiteEvo evolves from the neutral harness, averaged over three test runs. Initial is the neutral harness.

\Delta

is Evolved

-

Initial, before rounding. The GAIA run blocks known copies of the benchmark’s answers.

Harness	Mode	P@1	P@2	API cost ($)
Bare	–	0.0	0.0	–
Bare + format	–	0.0	0.0	–
HarnessX-Seed	–	21.7	32.8	–
HarnessX	Transductive	36.0	50.5	190.32
Neutral	–	0.0	0.0	–
LiteEvo	Inductive	34.5	48.6	14.56

Appendix

Table 14: Results on AppWorld’s test-challenge split, whose tasks need apps unseen in training (Qwen3.5-9B). Bare gives the agent only the task text, and Bare + format adds a minimal hint on the reply format. Neutral is LiteEvo’s starting harness, and HarnessX-Seed the handcrafted HarnessX-style one that we wrote for AppWorld. The evolved rows use the harness each run returns; HarnessX evolves transductively, on test-normal. P@1 and P@2 are pass@1 and pass@2 in %, averaged over three test runs. API cost is that of the evolution run.

Figure 13: LiteEvo reaches its best AppWorld harness from a zero-scoring start in six rounds, at 15.2

\times

lower API cost than HarnessX. One transductive run on AppWorld’s 168 test-normal tasks: pass@2 of each round’s harness, measured inside the loop (lines); the cumulative API cost of producing each round’s harness (bars); and the components each round added (

+

) or modified (

\sim

). The star marks the harness that LiteEvo returns, whose graded pass@2 over three test runs is 67.7 (Table 1 ). Our HarnessX reproduction starts from a hand-written AppWorld seed (Appendix C.5 ).

Benchmark	Mode	Analyst setup	# Analysts	pass@1 (%)	pass@2 (%)	Best Round	API cost ($)
ALFWorld	Transductive	Per-worker	4	96.6	98.0	7	19.26
	Transductive	GRP	1	92.6	96.3	5	8.24
	Inductive	Per-worker	4	92.5	96.3	6	20.05
	Inductive	GRP	1	90.3	94.8	4	9.17
WebShop	Transductive	Per-worker	4	51.8	62.7	1	24.78
	Transductive	GRP	1	54.0	62.7	5	9.00

Appendix

Table 15: Analyst setup under Compose (Appendix D.10 ; Qwen3.5-9B). Per-worker runs one analyst for each worker, on that worker’s own trajectories. GRP is failure-oriented group sampling: a single analyst reads grouped trajectories of all workers. Scores average three test runs, and API cost is that of one run.

Benchmark	Sampling	pass@1 (%)	pass@2 (%)	Best Round
ALFWorld	Default	92.6	96.3	5
	All-success	79.9	86.6	3
	All-fail	84.7	89.3	7
	Random	90.1	94.3	7
WebShop	Default	54.0	62.7	5
	All-success	49.5	62.0	7

Appendix

Table 16: Which task groups the analyst reads (Qwen3.5-9B, transductive): always 13, or 7 on SWE-bench Verified. Default reads 10 mixed-outcome groups and 3 all-fail groups (5 and 2 on SWE-bench Verified). All-success , All-fail and Random read groups that every worker solved, groups that no worker solved, or random groups, each filled up from the other groups when too few exist. Scores average three test runs.

Benchmark	Round	0	1	2	3	4	5	6	7	Total
ALFWorld	Add	4	4	3	2	1	2	1	2	19
	Update	0	0	0	0	1	0	0	1	2
	Merge	0	0	1	1	0	0	0	0	2
	Drop	0	0	1	0	0	0	0	0	1
	Picked / Size	4/4	8/8	10/10	12/12	13/13	15/15	14/16	16/18	–
WebShop	Add	3	2	2	1	1	3	2	0	14

Appendix

Table 17: Per-benchmark breakdown of Table 4 . Picked / Size is the number of components the composer selects over the number it can select from. GAIA counts are reconstructed from library snapshots.

Benchmark	Enforced Memory	pass@1 (%)	pass@2 (%)	Best Round
GAIA	None (default)	53.6	65.2	7
	Action book	45.1	55.8	5
	URL blacklist	46.1	58.4	7
AppWorld	None (default)	51.9	67.7	6
AppWorld	Login recipes	13.2	15.4	6

Appendix

Table 18: Pinning into every composition a hand-written memory that persists across episodes (Qwen3.5-9B, transductive). The memory reads and writes a store shared by episodes, and no episode reads entries from attempts on its own task. Action book stores a failed action with the next action that worked. URL blacklist stores, as an avoid-list, web domains that blocked access in at least two tasks. Login recipes stores the API calls that logged in. None rows are the default runs of Table 16 .

Benchmark	Merge	pass@1 (%)	pass@2 (%)
GAIA	None (default)	53.6	65.2
GAIA	Editor + Librarian	48.1	59.9
ALFWorld	None (default)	92.5	96.3
ALFWorld	Editor + Librarian	91.0	92.5
WebShop	None (default)	56.0	66.0
WebShop	Editor + Librarian	52.5	61.0

Appendix

Table 19: Merging the editor and the librarian into one meta-agent (Qwen3.5-9B). None keeps the two meta-agents separate: the editor selects and rewrites the analyst’s candidates, and the librarian applies them to the library. Editor + Librarian merges them and uses one analyst per worker. The None rows are the per-worker runs of Table 15 , except on GAIA, whose default is the group-sampling run of Table 16 . GAIA is transductive; the other benchmarks are inductive and scored on held-out tasks.

		ALFWorld		WebShop		AppWorld		GAIA		SWE-bench Verified		Avg.
Regime	Mode	P@1	P@2	P@1	P@2	P@1	P@2	P@1	P@2	P@1	P@2	P@1	P@2
Transductive	Refine	88.3	93.9	58.6	67.8	27.4	41.6	48.0	58.8	50.7	59.4	54.6	64.3
Transductive	Compose	96.6	98.0	51.8	62.7	40.3	54.6	49.8	63.4	54.4	61.8	58.6	68.1
Inductive	Refine	93.1	95.5	52.0	59.2	31.4	45.6	–	–	53.9	63.1	57.6	65.8
Inductive	Compose	92.5	96.3	56.0	66.0	49.4	65.3	–	–	52.7	61.8	62.7	72.4

Appendix

Table 20: Two ways for the composer to produce a harness (Qwen3.5-9B). Refine composes a harness after the first round and then edits it in every round; Compose builds a fresh harness from the library in every round. Both use one analyst per worker. P@1 and P@2 are pass@1 and pass@2 in %, averaged over three test runs. Bold marks the better mode within each regime. Avg. averages five benchmarks transductively and four inductively, since GAIA has no inductive run.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 185 papers

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16LLM Agent Evaluation Agent Harness Optimization
Rethinking the Evaluation of Harness Evolution for Agents
Jul 14, 2026Yike Wang, Huaisheng Zhu, Zhengyu Hu +8LLM Agent Evaluation Agent Harness Optimization
Allen Institute for AI · University of Washington · Independent
Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses
Aug 9, 2026Tailin ZhouAgent Harness Optimization LLM Agent Self-Improvement
HKUST

Sep 1, 2026J/K move · Enter open · S save

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei+15

LLM Agent Evaluation Agent Harness Optimization AI Agent Benchmarks

Open paper AskPDF arXiv

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Abstract

Agent Harness Optimization

Explore similar work

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Rethinking the Evaluation of Harness Evolution for Agents

Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses