Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Jens Lehmann; Sahar Vahdati

cs.AIOct 21, 2026

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Authors: Jens Lehmann, Sahar Vahdati

Organizations: Amazon∗ · Technische Universität Dresden · Universität Hannover · TIB – Leibniz Information Centre for Science and Technology and University Library

Abstract

A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.

Figures & tables

Panel of Figure 1 — Figure 1: Controlled harness-adaptation workflow. At each stage, the same model profile uses the fixed coding bootstrap to propose a candidate. The controller checks that it can execute and interact with two development games. A failed candidate may be repaired; a passing candidate is frozen as $H_{t}$ . Five-game development evidence from $H_{0}$ and $H_{1}$ seeds the next revision. $H_{2}$ runs on the same games for evaluation only. All snapshots are evaluated on twenty held-out games after adaptation.

Panel of Figure 2 — Figure 2: Two observations from FT09 (Level 1 on the left, and Level 7 on the right), a public ARC-AGI-3 game. The agent must infer that the small patterned tiles specify which neighboring tiles must match or differ from a reference color. Clicking cycles tile colors, which need to be matched to target patterns. The later levels introduce controls that change several tiles together. These rules must be discovered through interaction.

Figure 3: Construction strength varies, and development gains transfer unreliably. (a) Held-out condition means on a common scale. (b) Each condition has two points, averaging its three runs: initial-to-revision-1 (open) and initial-to-revision-2 (filled). Labels count strict-sign quadrants. Points below zero regress on held-out games. Triangles denote higher-effort conditions.

Figure 4: Separate outcomes reveal which harnesses improve, save cost, or trade one for the other. Each row identifies a model–effort condition and run. Bars show magnitudes of observed

H_{2}-H_{0}

changes in mean held-out RHAE and inference cost per game; headers specify the direction. Each metric uses a common linear scale across panels. Categories use unrounded changes.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Model profile	Reasoning effort	Context	Max output
GPT-6 Astra	high	1.05M	64K
GPT-5.6 Sol	high	1.05M	120K
GPT-5.6 Terra	high	1.05M	120K
GPT-5.6 Luna	high	1.05M	120K
Claude Fable 5	high	1.00M	120K
Claude Opus 5	high	1.00M	120K

Appendix

Table 1: Study conditions. Builder and runtime profiles are matched within each row. Context is the advertised window; max output is the configured per-response gameplay cap. Coding caps are given below. Exact deployment and pricing-profile hashes are retained in run protocols.

Work	Editable artifact and schedule	Final evidence	Main protocol difference
STOP	One optimizer function over 1–4 recursive depths	Fresh optimization tasks	Restricted to an optimizer-function artifact.
ADAS	Code-defined agent functions under archive-guided search	All discovered ARC agents on held-out test; five evaluations per agent	Searches agent functions; our study tracks prescribed revisions of complete interactive harnesses.
AFlow / GEPA	Workflows or prompts under budgeted search	Validation-selected candidate on test	Constrained artifact and selection over many candidates.
SICA / DGM	Complete coding-agent codebase with archive-guided evolution	Same-benchmark selection plus transfer	Whole-code evolution with variable lineages and update counts.
Meta-Harness / AHE	Executable harness source under multi-iteration score-and-trace search	Best or final frozen harness	Seeded candidate search with selected-harness outcomes.
RRSI ( Xia et al., 2026 )	Harness source; 20–40 rounds with regularized proposals and acceptance	Evolved harness on held-out tasks and OOD benchmarks	Tests a selection intervention under multi-candidate search, rather than every prescribed descendant.

Appendix

Table 2: Closest protocol precedents. Strict test means that test tasks do not influence proposals, acceptance, stopping, or candidate selection.

OpenAI
		RHAE / cost (USD/game)
Condition	Rep.	Initial	Revision 1	Revision 2
GPT-6 Astra high	1	97.14/11.90	93.77/10.86	97.55/10.24
	2	58.03/8.58	80.76/12.53	92.62/11.96
	3	93.59/18.77	93.23/11.87	92.94/12.92
GPT-5.6 Sol xhigh	1	43.95/31.39	18.49/4.89	21.99/7.18

Appendix

Table 3: All completed trajectories reported in the paper. Conditions are grouped by provider, with matched higher-effort profiles first. Logical costs use the publication price schedule and selected per-game token ledgers; three trajectories are recovered directly from those ledgers because final aggregate report assembly failed.

Condition	Initial	Revision 2	Change
Sol high	26.15	28.58	+2.43
Sol xhigh	41.98	41.94	-0.05
Opus high	11.71	15.43	+3.72
Opus xhigh	31.27	39.95	+8.68

Appendix

Table 4: Descriptive held-out means for matched reasoning-effort conditions. Each row contains three independent trajectories.

Deployment policy	RHAE	Gain	USD/game	$+/-/=$	$H_{0}/H_{1}/H_{2}$
Retain $H_{0}$	24.54	0.00	10.30	0/0/27	27/0/0
Deploy $H_{2}$	27.34	+2.80	7.57	15/12/0	0/0/27
Select by development RHAE	29.06	+4.53	9.15	13/5/9	9/6/12
Held-out oracle	30.66	+6.13	9.80	18/0/9	9/8/10

Appendix

Table 5: Post hoc checkpoint diagnostic. Development selection chooses the snapshot with highest five-game development RHAE , breaking ties toward the earlier snapshot. “

+/-/=

” counts held-out improvement, regression, and no change relative to

H_{0}

. Costs exclude outer-loop construction and selection expenditure. The held-out oracle is diagnostic only.

Snapshot	Mean USD/game	Median USD/game
Initial	10.30	9.18
Revision 1	7.15	6.26
Revision 2	7.57	7.18

Appendix

Table 6: Logical solving inference cost per held-out game across all 27 scored lineages. Logical cost includes replayed responses.

Transition	Pareto gain	Score $\uparrow$ /cost $\uparrow$	Cheaper/ RHAE $\downarrow$	Worse both	Tie/cost $\downarrow$	Tie/cost $\uparrow$
$H_{0}\!\to\!H_{1}$	6	9	9	2	0	1
$H_{1}\!\to\!H_{2}$	6	6	8	7	0	0
$H_{0}\!\to\!H_{2}$	9	6	9	3	0	0

Appendix

Table 7: Joint held-out score–cost movements. The “cheaper/ RHAE down” category records joint decreases in cost and RHAE .

Component	Initial	Revision 1	Revision 2	Total
Coding loop, 27 scored	607.92	371.78	372.64	1,352.34
Smoke qualification, 27 planned	408.02	111.11	127.72	646.84
Fresh selected-game attempts, 27 scored	6,749.95	4,712.60	4,667.55	16,130.09
List-price-equivalent total	7,765.89	5,195.49	5,167.90	18,129.27

Appendix

Table 8: Inference cost in USD under the frozen publication price schedule. Selected-game cost counts fresh model inference; logical replay charges used for score–cost comparison are excluded. Coding-bootstrap cost covers construction, revision, and repair; qualification is runtime inference used to select a smoke-qualified candidate.

$\epsilon$	Improve	Regress	Indeterminate	Pareto gain
0	15	12	0	9
1	11	8	8	7
5	9	3	15	5

Appendix

Table 9: Endpoint robustness to practical RHAE thresholds. Changes within

\pm\epsilon

are indeterminate; deployment Pareto gains also require lower solving cost.

Evaluation	RHAE	Actions	Levels completed	Complete games
0	53.56	8,492	102	8
1	46.86	8,509	94	6
2	43.38	9,382	90	6
3	48.21	9,389	92	7
4	45.32	8,515	91	6
5	55.37	8,459	104	8

Appendix

Table 10: Ten evaluations of one unchanged Sol-high

H_{2}

harness. Evaluation 0 is the original campaign observation. “Complete” counts games for which all levels were solved; RHAE also gives partial credit within the remaining games.

Stage	RHAE	Complete games	Levels completed	Actions
Initial	24.54	73/540 (13.5%)	34.5%	259.6
Revision 1	25.87	83/540 (15.4%)	36.4%	292.0
Revision 2	27.34	94/540 (17.4%)	37.1%	273.6

Appendix

Table 11: Held-out completion and action context across the same 540 evaluations per stage (27 lineages

\times

20 games). Level completion is the mean within-game fraction of completed levels. Actions are mean charged actions; lower action counts need not imply better efficiency because systems may stop without completing a game. These summaries do not isolate the components of RHAE .

Game	Split	$\Delta$ RHAE	$\Delta$ Complete	$\Delta$ Levels	$\Delta$ Actions
bp35	dev	+3.43	+3.7	+5.3	+46.5
ft09	dev	-0.45	+11.1	-2.5	-2.1
r11l	dev	+18.83	+14.8	+17.9	+22.1
tr87	dev	+8.11	+7.4	+11.1	+22.8
vc33	dev	+12.19	+11.1	+13.2	+76.1
ar25	heldout	-1.29	+3.7	-4.2	-80.7

Appendix

Table 12: Per-game initial-to-revision-2 changes, averaging the same 27 paired lineages. Completion columns are percentage-point changes in complete games and mean fraction of levels completed. Rows describe observed outcomes; no alternative feedback split was used to generate the harnesses.

Figure 5: Harnesses grow and undergo substantial revision without reliable gains. Generated Python counts model-produced source and tests and excludes the supplied llm_client.py . (a) All 27 lineages on a log scale, colored by initial-to-revision-2 held-out outcome; the black path is the stage median. (b) Each point is one selected parent-to-descendant Python diff. Churn is added plus removed lines, and points above zero improve held-out RHAE .

Condition	Initial harness family	$H_{0}\!\to\!H_{1}$ P/E/R	$H_{1}\!\to\!H_{2}$ P/E/R
Astra high	Persistent Python workers with policy reuse	0/0/3	0/1/2
Sol high	Direct multimodal batched planner	0/0/3	0/3/0
Terra high	Compact visual/text planner	0/0/3	0/2/1
Luna high	Direct model controller	0/2/1	0/2/1
Fable high	Strategist, executor, and replay system	1/0/2	1/0/2
Opus high	Layered world-model/planner hybrids	0/0/3	0/0/3

Appendix

Table 13: Strategy-transition counts across all conditions. Entries give Preserve/Extend/Reorient.

Condition	Replicate	$H_{0}\!\to\!H_{1}$	$H_{1}\!\to\!H_{2}$
Astra high	1	R	R
Astra high	2	R	R
Astra high	3	R	E
Sol high	1	R	E
Sol high	2	R	E
Sol high	3	R	E

Appendix

Table 14: Strategy transitions for all 27 replicates. P, E, and R follow the definitions above; Table 3 gives the corresponding score–cost trajectories.

Code group	Central question	Observable indicators
Evidence access	Does the updater inspect item-specific support before editing?	Feedback summaries, traces, logs, score breakdowns, errors, and reproducers ( Evid ).
Diagnosis	Does it connect cited evidence to a mechanism and prediction?	Pre-edit symptoms, localized mechanisms, predictions, counterfactuals, and boundary conditions ( Diag ).
Diagnostic probing	Does evidence discriminate among hypotheses?	Targeted inspection, stated alternatives, refutable expectations, and result-governed narrowing ( Probe ).
Plan and alignment	Are targets, actions, criteria, and retained edits linked?	Enacted plans, eligible-feedback coverage, and final-edit alignment ( Plan , Cov , Align ).
Change control	Does it contain risk and protect the incumbent?	Executable scope, fallbacks, baseline checks, comparison, narrowing, rollback, and no-change decisions ( Scope , Protect ).
Verification	Does it test the retained change and remain runnable?	Reproducers, targeted and regression tests, smoke repair, requalification, and proximal results ( Verify , Result ).

Appendix

Table 15: Episode codebook used for observable harness revision.

Code	$n$	Exact	$\kappa_{w}$	$\alpha$
Evid	54	1.00	–	–
Diag	30	.67	.35	.38
Probe	53	.85	.66	.67
Plan	27	.85	.70	.70
Cov	54	1.00	–	–
Align	54	1.00	–	–

Appendix

Table 16: Unadjudicated reliability under frozen codebook version 2.

n

excludes variable-specific missingness.

\kappa_{w}

uses quadratic disagreement weights

d_{ij}=((i-j)/(K-1))^{2}

for

K

ordered categories.

\alpha

uses nominal disagreement and exposes prevalence effects. Dashes denote undefined chance-corrected coefficients for invariant labels.

Code	$n$	$\kappa_{w}$	Contrast/counts	Interpretation
Scope containment	54	.96	$-3.94$	Scoped minus broad, interval $[-9.09,1.22]$ , $p=.12$ .
Post-result adaptation	54	.84	$-3.24$	Further-action minus handoff, interval $[-8.54,2.38]$ , $p=.174$ .

Appendix

Table 17: Exploratory process associations for labels with adequate inter-session reliability and outcome coverage. Contrasts are immediate held-out RHAE change at the highest minus lowest shown code. Intervals resample complete trajectories;

p

values permute trajectory profiles. Associations are descriptive.

Control	Verified result
Frozen source	All 81 original snapshot fingerprints and file counts match, covering 1,985 files.
Evaluated identity	All 2,025 selected evaluations match the corresponding snapshot, runtime model-profile fingerprint, environment fingerprint, and configured output cap.
Prompts and bounds	Each rendered prompt is constant across replicates within each of nine conditions; all 27 lineages share one resource-limit profile.
Hosted validation	All 81 validations are complete, cover 25 games, agree with recorded scores, match snapshot identity, and use the same trace identity as local validation.
Coding preservation	All 54 blinded packets match their manifests; their labels and rationales are unchanged from the hash-verified frozen coding passes and consensus.

Appendix

Table 18: Post-collection integrity checks against retained original records.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 387 papers

Sep 1, 2026

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16

LLM Agent Evaluation Agent Harness Optimization AI Agent Benchmarks

Ask PDF

May 28, 2026

Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

LLM agents are increasingly deployed as systems built around editable external harnesses, including prompts, skills, memories and tools, that shape task execution without changing model parameters. Harness self-evolution adapts such agents by updating these harnesses from execution evidence. Yet it remains unclear whether a model's base capability in task-solving predicts its capabilities in harness self-evolution: which models produce useful harness updates, and which actually benefit from them? We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving. Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains; even Qwen3.5-9B's updates yield gains comparable to those of Claude Opus~4.6. Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier. We trace low gains at the weak tier to two failure modes: weak-tier models may fail to activate relevant harness artifacts, or activate them but fail to follow them faithfully. These findings suggest investing capability budget in the task-solving agent rather than the evolver, and targeting harness invocation and long-horizon instruction following in agent training. Our source code is publicly available at https://github.com/A-EVO-Lab/a-evolve/tree/release/harness-evolution.

Minhua Lin, Juncheng Wu, Zijun Wang +14

1The Pennsylvania State University · 2UC Santa Cruz · 3Amazon +3

LLM Agent Evaluation Agent Harness Optimization LLM Agent Self-Improvement LLM Agent Harnesses Self-Evolving Agents

Ask PDF

Jun 8, 2026

Self-Harness: Harnesses That Improve Themselves

The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and rapidly evolving. In this paper, we introduce Self-Harness, a new paradigm in which an LLM-based agent improves its own operating harness, without relying on human engineers or stronger external agents. We operationalize Self-Harness as an iterative loop with three stages: Weakness Mining, which identifies model-specific failure patterns from execution traces; Harness Proposal, which generates diverse yet minimal harness modifications tied to these failures; and Proposal Validation, which accepts candidate edits only after regression testing. We instantiate Self-Harness on Terminal-Bench-2.0 using a minimal initial harness and three base models from diverse families: MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5. Across all three models, Self-Harness consistently improves performance, with held-out pass rates increasing from 40.5% to 61.9%, 23.8% to 38.1%, and 42.9% to 57.1%, respectively. Qualitative analyses further show that Self-Harness does not simply add generic instructions, but effectively turns model-specific weaknesses into concrete, executable harness changes. These results suggest a path toward LLM-based agents that are not merely shaped by their harnesses, but can also participate in reshaping them.

Hangfan Zhang, Shao Zhang, Kangcong Li +5

Shanghai Artificial Intelligence Laboratory

Agent Harness Optimization LLM Agent Self-Improvement LLM Agents LLM Agent Harnesses

Ask PDF

Abstract

LLM Agent Harnesses

Explore similar work

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

Self-Harness: Harnesses That Improve Themselves