Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Jens Lehmann; Sahar Vahdati

cs.AIOct 21, 2026

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Authors: Jens Lehmann, Sahar Vahdati

Organizations: Amazon∗ · Technische Universität Dresden · Universität Hannover · TIB – Leibniz Information Centre for Science and Technology and University Library

Abstract

A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.

Figures & tables

Panel of Figure 1 — Figure 1: Controlled harness-adaptation workflow. At each stage, the same model profile uses the fixed coding bootstrap to propose a candidate. The controller checks that it can execute and interact with two development games. A failed candidate may be repaired; a passing candidate is frozen as $H_{t}$ . Five-game development evidence from $H_{0}$ and $H_{1}$ seeds the next revision. $H_{2}$ runs on the same games for evaluation only. All snapshots are evaluated on twenty held-out games after adaptation.

Panel of Figure 2 — Figure 2: Two observations from FT09 (Level 1 on the left, and Level 7 on the right), a public ARC-AGI-3 game. The agent must infer that the small patterned tiles specify which neighboring tiles must match or differ from a reference color. Clicking cycles tile colors, which need to be matched to target patterns. The later levels introduce controls that change several tiles together. These rules must be discovered through interaction.

Figure 3: Construction strength varies, and development gains transfer unreliably. (a) Held-out condition means on a common scale. (b) Each condition has two points, averaging its three runs: initial-to-revision-1 (open) and initial-to-revision-2 (filled). Labels count strict-sign quadrants. Points below zero regress on held-out games. Triangles denote higher-effort conditions.

Figure 4: Separate outcomes reveal which harnesses improve, save cost, or trade one for the other. Each row identifies a model–effort condition and run. Bars show magnitudes of observed

H_{2}-H_{0}

changes in mean held-out RHAE and inference cost per game; headers specify the direction. Each metric uses a common linear scale across panels. Categories use unrounded changes.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Model profile	Reasoning effort	Context	Max output
GPT-6 Astra	high	1.05M	64K
GPT-5.6 Sol	high	1.05M	120K
GPT-5.6 Terra	high	1.05M	120K
GPT-5.6 Luna	high	1.05M	120K
Claude Fable 5	high	1.00M	120K
Claude Opus 5	high	1.00M	120K

Appendix

Table 1: Study conditions. Builder and runtime profiles are matched within each row. Context is the advertised window; max output is the configured per-response gameplay cap. Coding caps are given below. Exact deployment and pricing-profile hashes are retained in run protocols.

Work	Editable artifact and schedule	Final evidence	Main protocol difference
STOP	One optimizer function over 1–4 recursive depths	Fresh optimization tasks	Restricted to an optimizer-function artifact.
ADAS	Code-defined agent functions under archive-guided search	All discovered ARC agents on held-out test; five evaluations per agent	Searches agent functions; our study tracks prescribed revisions of complete interactive harnesses.
AFlow / GEPA	Workflows or prompts under budgeted search	Validation-selected candidate on test	Constrained artifact and selection over many candidates.
SICA / DGM	Complete coding-agent codebase with archive-guided evolution	Same-benchmark selection plus transfer	Whole-code evolution with variable lineages and update counts.
Meta-Harness / AHE	Executable harness source under multi-iteration score-and-trace search	Best or final frozen harness	Seeded candidate search with selected-harness outcomes.
RRSI ( Xia et al., 2026 )	Harness source; 20–40 rounds with regularized proposals and acceptance	Evolved harness on held-out tasks and OOD benchmarks	Tests a selection intervention under multi-candidate search, rather than every prescribed descendant.

Appendix

Table 2: Closest protocol precedents. Strict test means that test tasks do not influence proposals, acceptance, stopping, or candidate selection.

OpenAI
		RHAE / cost (USD/game)
Condition	Rep.	Initial	Revision 1	Revision 2
GPT-6 Astra high	1	97.14/11.90	93.77/10.86	97.55/10.24
	2	58.03/8.58	80.76/12.53	92.62/11.96
	3	93.59/18.77	93.23/11.87	92.94/12.92
GPT-5.6 Sol xhigh	1	43.95/31.39	18.49/4.89	21.99/7.18

Appendix

Table 3: All completed trajectories reported in the paper. Conditions are grouped by provider, with matched higher-effort profiles first. Logical costs use the publication price schedule and selected per-game token ledgers; three trajectories are recovered directly from those ledgers because final aggregate report assembly failed.

Condition	Initial	Revision 2	Change
Sol high	26.15	28.58	+2.43
Sol xhigh	41.98	41.94	-0.05
Opus high	11.71	15.43	+3.72
Opus xhigh	31.27	39.95	+8.68

Appendix

Table 4: Descriptive held-out means for matched reasoning-effort conditions. Each row contains three independent trajectories.

Deployment policy	RHAE	Gain	USD/game	$+/-/=$	$H_{0}/H_{1}/H_{2}$
Retain $H_{0}$	24.54	0.00	10.30	0/0/27	27/0/0
Deploy $H_{2}$	27.34	+2.80	7.57	15/12/0	0/0/27
Select by development RHAE	29.06	+4.53	9.15	13/5/9	9/6/12
Held-out oracle	30.66	+6.13	9.80	18/0/9	9/8/10

Appendix

Table 5: Post hoc checkpoint diagnostic. Development selection chooses the snapshot with highest five-game development RHAE , breaking ties toward the earlier snapshot. “

+/-/=

” counts held-out improvement, regression, and no change relative to

H_{0}

. Costs exclude outer-loop construction and selection expenditure. The held-out oracle is diagnostic only.

Snapshot	Mean USD/game	Median USD/game
Initial	10.30	9.18
Revision 1	7.15	6.26
Revision 2	7.57	7.18

Appendix

Table 6: Logical solving inference cost per held-out game across all 27 scored lineages. Logical cost includes replayed responses.

Transition	Pareto gain	Score $\uparrow$ /cost $\uparrow$	Cheaper/ RHAE $\downarrow$	Worse both	Tie/cost $\downarrow$	Tie/cost $\uparrow$
$H_{0}\!\to\!H_{1}$	6	9	9	2	0	1
$H_{1}\!\to\!H_{2}$	6	6	8	7	0	0
$H_{0}\!\to\!H_{2}$	9	6	9	3	0	0

Appendix

Table 7: Joint held-out score–cost movements. The “cheaper/ RHAE down” category records joint decreases in cost and RHAE .

Component	Initial	Revision 1	Revision 2	Total
Coding loop, 27 scored	607.92	371.78	372.64	1,352.34
Smoke qualification, 27 planned	408.02	111.11	127.72	646.84
Fresh selected-game attempts, 27 scored	6,749.95	4,712.60	4,667.55	16,130.09
List-price-equivalent total	7,765.89	5,195.49	5,167.90	18,129.27

Appendix

Table 8: Inference cost in USD under the frozen publication price schedule. Selected-game cost counts fresh model inference; logical replay charges used for score–cost comparison are excluded. Coding-bootstrap cost covers construction, revision, and repair; qualification is runtime inference used to select a smoke-qualified candidate.

$\epsilon$	Improve	Regress	Indeterminate	Pareto gain
0	15	12	0	9
1	11	8	8	7
5	9	3	15	5

Appendix

Table 9: Endpoint robustness to practical RHAE thresholds. Changes within

\pm\epsilon

are indeterminate; deployment Pareto gains also require lower solving cost.

Evaluation	RHAE	Actions	Levels completed	Complete games
0	53.56	8,492	102	8
1	46.86	8,509	94	6
2	43.38	9,382	90	6
3	48.21	9,389	92	7
4	45.32	8,515	91	6
5	55.37	8,459	104	8

Appendix

Table 10: Ten evaluations of one unchanged Sol-high

H_{2}

harness. Evaluation 0 is the original campaign observation. “Complete” counts games for which all levels were solved; RHAE also gives partial credit within the remaining games.

Stage	RHAE	Complete games	Levels completed	Actions
Initial	24.54	73/540 (13.5%)	34.5%	259.6
Revision 1	25.87	83/540 (15.4%)	36.4%	292.0
Revision 2	27.34	94/540 (17.4%)	37.1%	273.6

Appendix

Table 11: Held-out completion and action context across the same 540 evaluations per stage (27 lineages

\times

20 games). Level completion is the mean within-game fraction of completed levels. Actions are mean charged actions; lower action counts need not imply better efficiency because systems may stop without completing a game. These summaries do not isolate the components of RHAE .

Game	Split	$\Delta$ RHAE	$\Delta$ Complete	$\Delta$ Levels	$\Delta$ Actions
bp35	dev	+3.43	+3.7	+5.3	+46.5
ft09	dev	-0.45	+11.1	-2.5	-2.1
r11l	dev	+18.83	+14.8	+17.9	+22.1
tr87	dev	+8.11	+7.4	+11.1	+22.8
vc33	dev	+12.19	+11.1	+13.2	+76.1
ar25	heldout	-1.29	+3.7	-4.2	-80.7

Appendix

Table 12: Per-game initial-to-revision-2 changes, averaging the same 27 paired lineages. Completion columns are percentage-point changes in complete games and mean fraction of levels completed. Rows describe observed outcomes; no alternative feedback split was used to generate the harnesses.

Figure 5: Harnesses grow and undergo substantial revision without reliable gains. Generated Python counts model-produced source and tests and excludes the supplied llm_client.py . (a) All 27 lineages on a log scale, colored by initial-to-revision-2 held-out outcome; the black path is the stage median. (b) Each point is one selected parent-to-descendant Python diff. Churn is added plus removed lines, and points above zero improve held-out RHAE .

Condition	Initial harness family	$H_{0}\!\to\!H_{1}$ P/E/R	$H_{1}\!\to\!H_{2}$ P/E/R
Astra high	Persistent Python workers with policy reuse	0/0/3	0/1/2
Sol high	Direct multimodal batched planner	0/0/3	0/3/0
Terra high	Compact visual/text planner	0/0/3	0/2/1
Luna high	Direct model controller	0/2/1	0/2/1
Fable high	Strategist, executor, and replay system	1/0/2	1/0/2
Opus high	Layered world-model/planner hybrids	0/0/3	0/0/3

Appendix

Table 13: Strategy-transition counts across all conditions. Entries give Preserve/Extend/Reorient.

Condition	Replicate	$H_{0}\!\to\!H_{1}$	$H_{1}\!\to\!H_{2}$
Astra high	1	R	R
Astra high	2	R	R
Astra high	3	R	E
Sol high	1	R	E
Sol high	2	R	E
Sol high	3	R	E

Appendix

Table 14: Strategy transitions for all 27 replicates. P, E, and R follow the definitions above; Table 3 gives the corresponding score–cost trajectories.

Code group	Central question	Observable indicators
Evidence access	Does the updater inspect item-specific support before editing?	Feedback summaries, traces, logs, score breakdowns, errors, and reproducers ( Evid ).
Diagnosis	Does it connect cited evidence to a mechanism and prediction?	Pre-edit symptoms, localized mechanisms, predictions, counterfactuals, and boundary conditions ( Diag ).
Diagnostic probing	Does evidence discriminate among hypotheses?	Targeted inspection, stated alternatives, refutable expectations, and result-governed narrowing ( Probe ).
Plan and alignment	Are targets, actions, criteria, and retained edits linked?	Enacted plans, eligible-feedback coverage, and final-edit alignment ( Plan , Cov , Align ).
Change control	Does it contain risk and protect the incumbent?	Executable scope, fallbacks, baseline checks, comparison, narrowing, rollback, and no-change decisions ( Scope , Protect ).
Verification	Does it test the retained change and remain runnable?	Reproducers, targeted and regression tests, smoke repair, requalification, and proximal results ( Verify , Result ).

Appendix

Table 15: Episode codebook used for observable harness revision.

Code	$n$	Exact	$\kappa_{w}$	$\alpha$
Evid	54	1.00	–	–
Diag	30	.67	.35	.38
Probe	53	.85	.66	.67
Plan	27	.85	.70	.70
Cov	54	1.00	–	–
Align	54	1.00	–	–

Appendix

Table 16: Unadjudicated reliability under frozen codebook version 2.

n

excludes variable-specific missingness.

\kappa_{w}

uses quadratic disagreement weights

d_{ij}=((i-j)/(K-1))^{2}

for

K

ordered categories.

\alpha

uses nominal disagreement and exposes prevalence effects. Dashes denote undefined chance-corrected coefficients for invariant labels.

Code	$n$	$\kappa_{w}$	Contrast/counts	Interpretation
Scope containment	54	.96	$-3.94$	Scoped minus broad, interval $[-9.09,1.22]$ , $p=.12$ .
Post-result adaptation	54	.84	$-3.24$	Further-action minus handoff, interval $[-8.54,2.38]$ , $p=.174$ .

Appendix

Table 17: Exploratory process associations for labels with adequate inter-session reliability and outcome coverage. Contrasts are immediate held-out RHAE change at the highest minus lowest shown code. Intervals resample complete trajectories;

p

values permute trajectory profiles. Associations are descriptive.

Control	Verified result
Frozen source	All 81 original snapshot fingerprints and file counts match, covering 1,985 files.
Evaluated identity	All 2,025 selected evaluations match the corresponding snapshot, runtime model-profile fingerprint, environment fingerprint, and configured output cap.
Prompts and bounds	Each rendered prompt is constant across replicates within each of nine conditions; all 27 lineages share one resource-limit profile.
Hosted validation	All 81 validations are complete, cover 25 games, agree with recorded scores, match snapshot identity, and use the same trace identity as local validation.
Coding preservation	All 54 blinded packets match their manifests; their labels and rationales are unchanged from the hash-verified frozen coding passes and consensus.

Appendix

Table 18: Post-collection integrity checks against retained original records.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 387 papers

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16LLM Agent Evaluation Agent Harness Optimization
Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
May 28, 2026Minhua Lin, Juncheng Wu, Zijun Wang +14LLM Agent Evaluation Agent Harness Optimization
1The Pennsylvania State University · 2UC Santa Cruz · 3Amazon +3
Self-Harness: Harnesses That Improve Themselves
Jun 8, 2026Hangfan Zhang, Shao Zhang, Kangcong Li +5Agent Harness Optimization LLM Agent Self-Improvement
Shanghai Artificial Intelligence Laboratory

Sep 1, 2026J/K move · Enter open · S save

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei+15

LLM Agent Evaluation Agent Harness Optimization AI Agent Benchmarks

Open paper AskPDF arXiv

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Abstract

LLM Agent Harnesses

Explore similar work

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

Self-Harness: Harnesses That Improve Themselves