How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

Kirill Brilliantov; Alejandro Hernández-Cano; Emmanuel Abbé

cs.AISep 30, 2026

How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

Authors: Kirill Brilliantov, Alejandro Hernández-Cano, Emmanuel Abbé

Organizations: EPFL · Apple

Abstract

Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orchestrators, dedicated retrieval subagents, and more. While such harnesses expand, the use of more primitive but improved coding agents - where LLMs have direct access to the execution environment through read, write, and bash primitives - has received little attention in the field. In this paper we find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance. Via a series of large-scale systematic ablation studies, we argue that the machinery layers become redundant in the coding agent setting. We conclude that the effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.

Figures & tables

Figure 1 : Performance across backbones and harnesses. Any-medal rate (

\uparrow

, top) and mean percentile (

\uparrow

, bottom), self-select (filled) vs. oracle (open), macro-averaged over the full fixed30 set at 24h time budget. Brackets are 95% bootstrap CIs. Backbones are grouped along a single shared x-axis; color identifies the harness.

Figure 2 : Impact of a coding agent environment. Average percentile and medal rate obtained by Chat and Oneshot iterations under different LLM backbones in the default29 task set. Circles represent single-iteration performance, while squares indicate Best-of-N performance at six-hour budgets.

	Percentile ( $\uparrow$ )		Medal rate (%, $\uparrow$ )
	Self-select	Oracle	Self-select	Oracle
base	66.51 [62.46, 69.97]	69.29 [65.82, 72.13]	55.7 [48.2, 62.9]	60.4 [53.2, 66.8]
+D	63.62 [59.27, 68.26]	67.40 [63.22, 71.98]	45.2 [38.1, 54.8]	47.0 [38.1, 56.0]
+P3	67.77 [64.58, 70.62]	71.79 [69.34, 73.83]	52.4 [42.8, 61.9]	60.7 [53.6, 67.3]
+B+P3	58.91 [53.50, 64.05]	68.57 [65.77, 71.14]	33.3 [23.8, 40.5]	53.0 [45.8, 60.1]

Table 1 : Performance of Malena under different ablations on the fixed14 split at 24h budget. The 95% CI for each metric is reported in brackets. + indicates interventions on top of base , - indicates intervention removals.

	Surpassed SOTA (%) $\uparrow$
Backbone	Malena	AiScientist	MLEvolve
GLM-5.2	21.7 [15.0, 27.5]	17.5 [12.5, 22.5]	10.8 [7.5, 15.0]
Kimi-K3	26.7 [20.0, 32.5]	27.1 [20.0, 35.0]	11.7 [7.5, 15.0]

Table 2 : NatureBench results. Surpassed-SOTA rate (

\uparrow

, % of tasks where the oracle-best run surpasses the published SOTA), macro-averaged over tasks. A run with no valid submission counts as a failure. Brackets are 95% CIs.

Figure 3 : Left: oracle best-so-far percentile vs. wall-clock time, macro-averaged over fixed30 tasks, for different harnesses. Right: per-run distribution over all seeds of cumulative input, output and modelled cache-read tokens by the end of each run, plus a fourth group giving the resulting per-run USD cost (right-hand axis). Each box’s solid black line indicates median; dashed line indicates mean. MLEvolve’s cache is always exactly 0, since its tree-search nodes are independent LLM calls. All results shown are at the GLM 5.2 backbone.

Figure 4 : Per-technique rarity by method . Rarity of a technique is computed once per task over every method/backbone group’s runs combined (see Appendix E.2 ).

Figure 5 : Category composition of techniques used so far, over the course of a run , one column per method. The x-axis is checkpoint fraction through the run (

\in[0,1]

); the y-axis is the decayed share of technique-category tags at that point (stacked, summing to 1). Each technique contributes to a checkpoint’s composition with exponential decay by recency, giving a balanced view of the solution’s techniques that still favours ones added recently. Per-task and alternate half-life versions are in Appendix E.3 .

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

	Full competition ID	Split
alaska2* † ‖	alaska2-image-steganalysis	Medium
aptos2019	aptos2019-blindness-detection	Lite
billion-word*	billion-word-imputation	Medium
cassava* †	cassava-leaf-disease-classification	Medium
champs* † ‡	champs-scalar-coupling	Medium
freesound* †	freesound-audio-tagging-2019	Medium

Appendix

Table 3 : Full MLE-bench task suite used in the paper. Competitions marked with ‖ are not part of the default29 split. Tasks with * indicate the fixed14 task subset, and † indicates fixed10 . Tasks with ‡ were identified to have breaking issues, and results aggregated on them in fixed subsets come from our fixed version of such tasks.

s41467-025-65557-7	s41592-025-02662-x	s42256-023-00630-8	s42256-025-01010-0
s41551-024-01257-9	s41592-025-02826-9	s42256-023-00636-2	s42256-025-01026-6
s41551-024-01312-5	s41592-025-02886-x	s42256-023-00654-0	s43588-024-00689-2
s41587-024-02414-w	s42256-022-00447-x	s42256-023-00712-7	s43588-024-00698-1
s41592-023-01940-w	s42256-022-00459-7	s42256-024-00790-1	s43588-024-00716-2
s41592-023-02035-2	s42256-022-00518-z	s42256-024-00795-w	s43588-024-00732-2
s41592-023-02124-2	s42256-022-00526-z	s42256-024-00815-9	s43588-024-00757-7

Appendix

Table 4 : NatureBench task cohort used in the paper. Task IDs are the paper’s Nature-family DOI suffix.

Figure 6 : Impact of per-iteration time limits. Average percentile and effective-time obtained by Oneshot iterations under different per-iteration time budgets in the default29 set. Circled points represent single-iteration performance, while boxes indicate Best-of-N performance at six hour budgets. Error bars represent 95% CIs across our fixed 29-task set. The effective time measures the time taken for the run to finish. Best-of-N aggregates may remain below the target of 6h in the case of failing iterations that would exceed the maximum budget allocated.

Oneshot search strategy
	Percentile ( $\uparrow$ )	Medal rate (%, $\uparrow$ )
	$\Delta$	$\Delta$	won A/B/=
Best-of-N vs UCB1	-2.71 [-6.60, +1.44]	-5.3 [-11.8, +1.1]	4/7/18
Best-of-N vs Greedy	-1.98 [-5.86, +1.89]	-5.3 [-11.2, +0.6]	3/7/19
Best-of-N vs Chain	-0.51 [-4.66, +3.27]	-4.2 [-11.2, +2.9]	4/7/18
UCB1 vs Greedy	+0.73 [-2.85, +4.35]	+0.0 [-5.7, +5.7]	5/5/19

Appendix

Table 5 : Pairwise paired-by-task comparison. Results reported on the 24h budget, using the default29 task split. In an A vs B comparison, positive percentile differences indicate better average performance of A. Percentile difference reported along with 95% CIs.

	Percentile ( $\uparrow$ )		Selection gap ( $\downarrow$ )
	Self-select	Oracle
Malena	69.30 [66.74, 71.78]	71.18 [69.02, 73.46]	1.876 [0.805, 3.205]
Best-of-N	64.21 [61.19, 67.07]	71.04 [69.49, 72.66]	6.832 [4.336, 9.634]
UCB1	66.92 [64.30, 69.36]	70.89 [68.45, 73.35]	3.973 [2.484, 5.367]
Greedy	66.19 [63.43, 68.71]	68.98 [67.32, 70.56]	2.788 [1.011, 5.054]
Chain	64.72 [61.97, 67.42]	66.87 [64.08, 69.57]	2.152 [1.409, 2.899]

Appendix

Table 6 : Selection gap for different methods. Results reported on the 24h budget, with the default29 split, along with 95% CIs.

	Percentile ( $\uparrow$ )		Selection gap ( $\downarrow$ )
	Self-select	Oracle
BI	61.24 [58.85, 63.82]	63.26 [61.28, 65.38]	2.024 [1.125, 2.901]
base	62.56 [60.34, 64.63]	67.99 [66.74, 69.26]	5.433 [3.681, 7.359]
+V	63.74 [59.93, 67.04]	67.57 [66.23, 68.69]	3.833 [0.936, 7.263]
HV	61.55 [59.73, 64.36]	66.03 [63.38, 68.10]	4.479 [2.412, 5.944]

Appendix

Table 7 : Selection gap mitigation strategies. Performance reported with 95% CIs. default29 split at 12h budget.

Figure 7 : Behaviour of Malena under different time budgets. Average oracle percentile obtained by Malena on the default29 split with a DeepSeek V4 Pro (Preview) backbone model. Trajectories are extracted from the full run by scoring the submissions available at each time. Dotted lines are shown after a timeout target is hit, and maintain the performance achieved.

Figure 8 : Malena model ablation. Average percentile and medal rate obtained by the Malena iteration and Best-of-N Oneshot iterations at the 24h time budget in the default29 set. Error bars represent 95% CIs across our fixed 29-task set.

	Percentile ( $\uparrow$ )	Medal rate (%, $\uparrow$ )
	$\Delta$	$\Delta$	won A/B/=
Gemma 4 31B	-3.97 [-7.98, -0.35]*	+7.9 [+4.3, +11.8]*	4/1/24
DeepSeek V4 Flash	+1.15 [-5.42, +7.77]	+3.3 [-6.3, +12.4]	9/6/14
DeepSeek V4 Pro	+9.39 [+3.36, +15.98]*	+12.8 [+2.9, +22.4]*	11/4/14
GLM 5.2	+5.52 [+1.93, +9.57]*	+12.0 [+5.7, +18.4]*	12/2/15
Kimi K3	+4.51 [+0.83, +8.96]*	+10.5 [+4.3, +18.1]*	8/4/17

Appendix

Table 8 : Pairwise paired-by-task comparison of different backbones. Reported is the performance difference between Malena (A) vs Best-of-N (B); higher numbers indicate stronger average Malena performance. Results reported using the default29 task split, with 95% CIs.

Infrastructure ablation (24h, 29 tasks)
	Percentile ( $\uparrow$ )		Medal rate (%, $\uparrow$ )
	Self-select	Oracle	Self-select	Oracle
base	69.30 [66.74, 71.78]	71.18 [69.02, 73.46]	59.4 [54.8, 63.9]	62.5 [58.4, 66.3]
+N	63.85 [59.73, 67.77]	65.98 [62.05, 69.83]	48.9 [43.1, 54.0]	52.3 [46.6, 57.5]
-J	70.47 [66.89, 74.25]	72.84 [69.55, 76.10]	62.1 [56.9, 67.2]	62.1 [56.9, 67.2]
-S-J	65.33 [62.05, 68.61]	68.36 [65.16, 71.45]	50.6 [44.8, 56.3]	56.9 [50.6, 62.6]

Appendix

Table 9 : Malena infrastructure ablation. Performance reported with 95% CIs. Top set: default29 split at 24h. Bottom set: fixed14 split at 12h.

Infrastructure ablation (24h, 29 tasks)
	Percentile ( $\uparrow$ )	Medal rate (%, $\uparrow$ )
	$\Delta$	$\Delta$	won A/B/=
base vs +N	+5.46 [+0.70, +10.67]*	+10.6 [+3.7, +17.9]*	8/1/20
base vs -J	-1.17 [-5.64, +3.34]	-2.6 [-10.2, +4.7]	3/6/20
base vs -S-J	+3.97 [-0.07, +7.97]	+8.9 [+1.7, +15.9]*	9/3/17
Orchestration ablation (24h, 14 tasks)

Appendix

Table 10 : Malena pairwise paired-by-task comparison. In an A vs B comparison, positive percentile differences indicate better average performance of A. Differences reported along with 95% CIs. Top set: 24h, default29 split. Middle set: 24h, fixed14 split. Bottom set: 12h, fixed14 split.

Task	alaska2	cassava	champs	freesound	hubmap	imet	kuzushiji	multi-modal	nfl	petfinder
Avg	–	1.2	–	1.5	–	2.0	4.2	12.5	2.5	1.7

Appendix

Table 11 : Avg. compactions before first medal.

	Self-selected		Oracle
Group	Any-medal	Mean percentile	Any-medal	Mean percentile
Malena 1M	57.5% [42.2, 71.5]	77.1 [68.0, 86.2]	65.0% [49.5, 77.9]	78.8 [69.8, 87.8]
Malena 64k	50.0% [35.2, 64.8]	73.5 [65.3, 81.7]	50.0% [35.2, 64.8]	74.5 [66.3, 82.7]

Appendix

Table 12 : Malena under a shrunk context/output budget. Any-medal rate (

\uparrow

) and mean percentile (

\uparrow

), with 95% CIs, self-selected and oracle (best-of-pool), on the full fixed10 task subset (10 tasks

\times

4 seeds,

n=40

per arm).

	Percentile ( $\uparrow$ )	Medal rate ( $\uparrow$ )
base	55.15 [54.31, 55.98]	36.8 [35.5, 38.2]
+S+J	56.58 [54.78, 58.41]	37.9 [34.5, 41.4]
+S+J+N	55.99 [54.18, 57.85]	37.1 [33.6, 40.1]
+H	57.08 [55.39, 58.80]	38.8 [35.8, 41.8]
+D	56.95 [55.12, 58.81]	39.2 [35.8, 42.7]
+V	54.47 [52.62, 56.38]	35.9 [32.4, 39.4]

Appendix

Table 13 : Oneshot infrastructure ablation. Average performance observed per Oneshot iteration reported with 95% CIs in the default29 split. Top set: GLM 5.2 backbone. Bottom set: DeepSeek V4 Pro (Preview) backbone.

GLM 5.2
	Percentile ( $\uparrow$ )	Medal rate (%, $\uparrow$ )
	$\Delta$	$\Delta$	won A/B/=
base vs +S+J	-1.43 [-3.29, +0.57]	-1.1 [-4.7, +2.7]	9/10/10
base vs +S+J+N	-0.85 [-2.88, +1.22]	-0.2 [-3.9, +3.2]	9/10/10
base vs +H	-1.93 [-3.85, -2.9e-03]*	-2.0 [-5.4, +1.3]	7/10/12
base vs +D	-1.80 [-3.82, +0.23]	-2.4 [-6.0, +1.2]	9/10/10

Appendix

Table 14 : Pairwise paired-by-task comparison. Average per-iteration difference observed. Results reported using the default29 task split. In an A vs B comparison, positive percentile differences indicate better average performance of A. Percentile difference reported along with 95% CIs. Top set: GLM 5.2 backbone. Bottom set: DeepSeek V4 Pro (Preview) backbone.

Kimi K3
	Percentile ( $\uparrow$ )		Medal rate ( $\uparrow$ )
	Self-select	Oracle	Self-select	Oracle
Malena	72.75 [70.75, 74.82]	75.54 [73.90, 77.35]	60.0 [55.6, 64.4]	65.6 [60.0, 70.0]
Arbor	68.46 [66.63, 70.08]	69.53 [67.76, 71.25]	60.0 [56.7, 63.3]	61.1 [57.8, 64.4]
AiScientist	64.15 [61.26, 67.10]	65.48 [62.54, 68.32]	52.2 [46.7, 57.8]	53.3 [47.8, 58.9]
MLEvolve	63.89 [62.00, 65.74]	70.32 [68.98, 71.58]	50.0 [46.7, 54.4]	63.3 [60.0, 66.7]

Appendix

Table 15 : MLE-bench performance with different harnesses and backbones. Any-medal rate (

\uparrow

) and mean percentile (

\uparrow

), self-selected vs. oracle (best-in-pool), macro-averaged over the full fixed30 set (mean over seeds within a task, then mean over tasks) at 24h time budget. A run with no valid submission counts as the worst possible outcome (no medal, worst percentile) rather than being excluded from its row’s average. Brackets are 95% bootstrap CIs (seed-resampling, task set fixed). Sections are grouped by backbone model; bold / underline / italic rank the 1st/2nd/3rd best value per column within a section, among rows on that section’s shared task set.

Kimi K3
	Percentile ( $\uparrow$ )	Medal rate (%, $\uparrow$ )
	$\Delta$	$\Delta$	won A/B/=
Malena vs Arbor	+4.29 [+1.54, +7.05]*	+0.0 [-5.6, +5.6]	5/5/20
Malena vs AiScientist	+8.60 [+4.91, +12.15]*	+7.8 [+0.0, +15.6]	7/2/21
Malena vs MLEvolve	+8.85 [+6.11, +11.70]*	+10.0 [+3.3, +16.7]*	7/2/21
GLM 5.2

Appendix

Table 16 : Paired Malena vs. baseline comparison across all models, macro-averaged per task, over the fixed30 task set. Higher percentile numbers indicate stronger average performance for Malena.

reasoning_effort	Any-medal rate (%) $\uparrow$ [95% CI]	None-rate (%) $\downarrow$	Mean percentile $\uparrow$ [95% CI]	Median reasoning (chars) $\downarrow$	Mean nodes/run $\uparrow$	Buggy rate (%) $\downarrow$
max	0.0 [0.0, 0.0]	13.8	42.6 [36.0, 48.5]	37124	18.5	61
medium (canonical)	25.0 [10.0, 30.0]	2.5	53.1 [44.9, 59.8]	7069	62.8	62

Appendix

Table 17 : MLEvolve on DeepSeek V4 Flash, max vs. medium reasoning effort , restricted to an identical 10-task subset and 24h timeout for both arms. Any-medal rate and none-rate are self-selected outcomes; mean percentile is oracle (best-in-pool). 95% CIs are bootstrapped over per-task run lists (2000 resamples), matching the convention used for the paper’s headline tables. Reasoning length is a per-LLM-call median pooled across all calls in each arm’s runs; node counts (mean/run) and the buggy-node rate are from each run’s full search-tree journal.

Configuration	Any-medal rate (%) $\uparrow$ [95% CI]	Gold (%)	Silver (%)	Bronze (%)	Mean percentile $\uparrow$ [95% CI]
resource_smart_llm (canonical)	28.3 [10.0, 50.0]	10.0	13.3	5.0	66.6 [52.9, 80.7]
resource_smart_policy	20.0 [0.0, 40.0]	2.5	10.0	7.5	52.2 [39.6, 65.5]

Appendix

Table 19 : ScienceFlow (DeepSeek V4 Flash), resource-admission ablation , identical 10-task subset and 24h timeout as Appendix D.3 ’s MLEvolve ablation. Both any-medal and mean percentile are oracle (best-of-pool) across every candidate ScienceFlow harvests per run, flipped so higher is better, matching every other percentile column in this paper. 95% CIs are bootstrapped over per-task run lists (2000 resamples), matching the convention used for the paper’s headline tables (Section 5.1 ).

Ablation	Baseline any-medal rate (%) $\uparrow$ [95% CI]	Ablation any-medal rate (%) $\uparrow$ [95% CI]
Executor timeout: 4h $\to$ 12h	12.5 [0.0, 50.0]	12.5 [0.0, 50.0]
Convergence early-stop: stop_after 8 $\to$ 999	50.0 [0.0, 100.0]	43.8 [25.0, 75.0]

Appendix

Table 20 : Arbor (GLM 5.2), executor-timeout and convergence ablations against the canonical 4-seed baseline, matched per task. Any-medal is self-selected, best-of-pool across every scored idea-tree node. 95% CIs are bootstrapped over per-task run lists (2000 resamples), matching the convention used for the paper’s headline tables.

Figure 9 : Left: oracle best-so-far vs. wall-clock time at Kimi K3 for Malena, Arbor, AiScientist, and MLEvolve. Right: per-run distribution of cumulative input, output(+reasoning), and modeled cache-read tokens, plus a USD-cost group at Kimi K3 pricing, for Malena, Arbor, AiScientist, and MLEvolve. MLEvolve’s cache group is omitted (always exactly 0, same reason as the GLM 5.2 figure).

Figure 10 : Oracle best-so-far vs. wall-clock time, faceted by task cluster. Malena’s lead is largest on cluster 3, smaller but present on clusters 2 and 4, and narrowest on cluster 1, where every method approaches ceiling performance by 24h.

	Malena	AiScientist	MLEvolve
p25	8	7	2
p50 (median)	16	18	8
p75	36	96	18

Appendix

Table 21 : Number of /evaluate calls per run on NatureBench. 25th/50th/75th percentiles across our full NatureBench run cohort, pooling both the GLM 5.2 and Kimi K3 backbones.

Figure 11 : Pooled category-share-decay curve (all labeled tasks) at half-lives 0.1, 0.15, and 0.2 (rows, top to bottom) – smaller half-life is a more local, noisier snapshot of recently-introduced techniques. As in Figure 5 , the x-axis is checkpoint fraction through the run (

\in[0,1]

) and the y-axis is the decayed share of technique-category tags at that point.

Figure 12 : Category-share-decay curve on aptos2019 alone, at half-lives 0.1, 0.15, and 0.2 (rows, top to bottom). As in Figure 5 , the x-axis is checkpoint fraction through the run (

\in[0,1]

) and the y-axis is the decayed share of technique-category tags at that point.

Figure 13 : Category-share-decay curve on nfl alone, at half-lives 0.1, 0.15, and 0.2 (rows, top to bottom). As in Figure 5 , the x-axis is checkpoint fraction through the run (

\in[0,1]

) and the y-axis is the decayed share of technique-category tags at that point.

Figure 14 : Technique-category composition of each method’s technique repertoire, per task (backbones pooled, completions excluded). The x-axis is method (Malena, AiScientist, MLEvolve, Oneshots); the y-axis is share of category-tags. Bar segments are shares of category-tags (a technique tagged with multiple categories counts toward each), not shares of techniques.

metric	group	p50	p90	p95	p99
Dolos containment	real-vs-real	0.008	0.077	0.143	0.453
Dolos containment	Malena-vs-real	0.002	0.009	0.013	0.020
mean-pooling cosine	real-vs-real	0.735	0.877	0.909	0.968
mean-pooling cosine	Malena-vs-real	0.773	0.880	0.896	0.919
Chamfer	real-vs-real	0.634	0.741	0.778	0.882
Chamfer	Malena-vs-real	0.638	0.713	0.726	0.746

Appendix

Table 22 : Contamination-check similarity scores by percentile: Malena-generated code vs. real fixed30 solutions, compared against a real-vs-real cross-label null baseline. Top: Dolos code-similarity containment. Bottom: bag-of-chunk-embedding similarity.

Explore similar work

Apr 28, 2026cs.CL

Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE), a closed loop that addresses these challenges through three matched observability pillars: (1) component observability gives every editable harness component a file-level representation so the action space is explicit and revertible; (2) experience observability distills millions of raw trajectory tokens into a layered, drill-down evidence corpus that an evolving agent can actually consume; and (3) decision observability pairs every edit with a self-declared prediction, later verified against the next round's task-level outcomes. Together, these pillars turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error. Empirically, ten AHE iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI (71.9%) and the self-evolving baselines ACE and TF-GRPO. The frozen harness transfers without re-evolution: on SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains across three alternate model families, indicating the evolved components encode general engineering experience rather than benchmark-specific tuning. Ablations localize the gain to tools, middleware, and long-term memory rather than the system prompt, suggesting factual harness structure transfers while prose-level strategy does not.

Jiahang Lin, Shichun Liu, Chengjun Pan +8

Fudan University · Shanghai Qiji Zhifeng Co., Ltd · Peking University

Agent Harness Coding Agents Auto Research Partial Observability Action Space

Ask PDF

May 18, 2026cs.CL

Code as Agent Harness

Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering. In emerging agentic systems, code is no longer only a target output. It increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification. We frame this shift through the lens of agent harnesses and introduce code as agent harness: a unified view that centers code as the basis for agent infrastructure. To systematically study this perspective, we organize the survey around three connected layers. First, we study the harness interface, where code connects agents to reasoning, action, and environment modeling. Second, we examine harness mechanisms: planning, memory, and tool use for long-horizon execution, together with feedback-driven control and optimization that make harness reliable and adaptive. Third, we discuss scaling the harness from single-agent systems to multi-agent settings, where shared code artifacts support multi-agent coordination, review, and verification. Across these layers, we summarize representative methods and practical applications of code as agent harness, spanning coding assistants, GUI/OS automation, embodied agents, scientific discovery, personalization and recommendation, DevOps, and enterprise workflows. We further outline open challenges for harness engineering, including evaluation beyond final task success, verification under incomplete feedback, regression-free harness improvement, consistent shared state across multiple agents, human oversight for safety-critical actions, and extensions to multimodal environments. By centering code as the harness of agentic AI, this survey provides a unified roadmap toward executable, verifiable, and stateful AI agent systems.

Xuying Ning, Katherine Tieu, Dongqi Fu +39

Agent Harness Coding Agents Repository-Level Code Understanding Multi-Agent Communication

Ask PDF

Sep 17, 2026cs.AI

An Empirical Study of Harness Design for Coding Agents

Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.

Run-Ze Fan, Zihao Zhang, Simin Ma +6

UMass Amherst · Emory University · UNC Charlotte

Coding Agents Agent Harness Empirical Study Token Budget Allocation

Ask PDF