Meta-Harness: End-to-End Optimization of Model Harnesses

Yoonho Lee; Roshen Nair; Qizheng Zhang; Kangwook Lee; Omar Khattab; Chelsea Finn

cs.AIMar 30, 2026

Meta-Harness: End-to-End Optimization of Model Harnesses

Authors: Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn

Organizations: Stanford · KRAFTON · MIT

Abstract

The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.

Figures & tables

Figure 1: (Left) On text classification, Meta-Harness outperforms the best prior hand-designed harnesses (ACE) and existing text optimizers (TTT-Discover, OpenEvolve), matching the next-best method’s final accuracy after just 4 evaluations. (Right) On TerminalBench-2, Meta-Harness outperforms all reported Claude Haiku 4.5 harnesses.

Figure 2: Meta-Harness search loop. (1) An agent reads a filesystem containing all prior candidates’ source code, execution traces, and scores, and proposes a new harness. (2) We evaluate the proposed harness on evaluation tasks. (3) All logs (proposed code, reasoning traces, evaluation scores) are stored in the filesystem in a new directory, and the loop repeats.

Method	History	Log content	MTok/iter
OPRO [ 51 ]	Window	past (solution, score) pairs	$0.002$
TextGrad [ 53 ]	Last	textual feedback on current artifact	$0.015$
AlphaEvolve [ 35 ]	Window	program database + eval. scores	$0.022$
GEPA [ 1 ]	Summary	reflective feedback from rollout traces	$0.008$
Feedback Descent [ 26 ]	Summary	comparison + textual feedback	$0.012$
TTT-Discover [ 55 ]	Window	prev. solution fragment	$0.026$

Table 1: Comparison of text optimization methods and their settings. Each row represents a method collapsed across tasks. Mtok/iter is our best estimate of the full context generated from one evaluation of a text artifact in the largest setting considered in each paper . This paper considers settings that yield orders-of-magnitude more context per artifact evaluation.

Method	Scores	Code	Summ.	Traces	Median $\uparrow$	Best Acc $\uparrow$	$>$ ZS
Scores Only	✓	✓	×	×	34.6	41.3	26
Scores + Summary	✓	✓	✓	×	34.9	38.7	23
Meta-Harness (full)	✓	✓	-	✓	50.0	56.7	39

Table 3: Ablation of the information available to the proposer in online text classification.

>

ZS: number of runs whose accuracy exceeded the zero-shot baseline. The full Meta-Harness interface substantially outperforms scores-only and scores-plus-summary ablations. Access to raw execution traces is the key ingredient for enabling harness search.

Method	Median	Best
GEPA [ 1 ]	32.6	40.2
Best-of-N	34.0	44.2
OpenEvolve [ 43 ]	39.1	43.3
TTT-Discover [ 54 ]	34.1	45.6
Meta-Harness	50.0	56.7

Table 4: Text classification accuracies of the harnesses proposed by different text optimizers (search set). Meta-Harness is substantially more effective at harness optimization.

Harness	SciC	FiNER	Amz5	FPB	GoEmo	Bank77	News	SciT	TwHate	Avg Acc	Ctx $\downarrow$
Zero-shot	32.7	56.0	52.7	90.0	42.0	80.7	84.7	89.3	75.3	67.0	-
Few-shot (8)	34.0	63.0	54.0	90.0	44.0	82.7	84.7	91.3	76.7	68.9	2.2
Few-shot (32)	38.7	62.0	53.3	90.7	43.3	86.0	85.3	90.7	76.7	69.6	5.2
Few-shot (all)	35.3	61.0	50.0	93.3	42.7	80.7	84.0	90.0	76.7	68.2	7.4
ACE [ 59 ]	40.7	74.0	48.0	96.7	44.0	83.3	86.0	90.7	68.7	70.2	11.7
Meta-Harness	53.3	67.0	60.0	94.0	46.0	82.7	86.7	91.3	77.3	73.1	7.3

Table 5: OOD text classification dataset evaluation. We report test accuracy for each dataset and the average additional context tokens across all nine datasets. Meta-Harness outperforms the next best method by 2.9 points on these 9 previously unseen tasks.

Method	GPT-5.4n	GPT-5.4m	Gem-3.1FL	Gem-3F	GPT-20B	Avg.
No Retriever	23.0	28.8	28.6	42.6	47.6	34.1
Dense Retrieval ( $k{=}1$ )	27.1 ( +4.1 )	24.5 ( -4.3 )	31.3 ( +2.7 )	42.3 ( -0.3 )	46.9 ( -0.7 )	34.4 ( +0.3 )
Dense Retrieval ( $k{=}5$ )	31.1 ( +8.1 )	28.3 ( -0.5 )	37.1 ( +8.5 )	47.2 ( +4.6 )	46.7 ( -0.9 )	38.1 ( +4.0 )
Random Few-shot	23.1 ( +0.1 )	24.5 ( -4.3 )	31.0 ( +2.4 )	40.4 ( -2.2 )	41.8 ( -5.8 )	32.2 ( -1.9 )
BM25 Retrieval	30.2 ( +7.2 )	29.2 ( +0.4 )	32.8 ( +4.2 )	46.6 ( +4.0 )	48.9 ( +1.3 )	37.5 ( +3.4 )
Meta-Harness	31.7 ( +8.7 )	30.4 ( +1.6 )	34.9 ( +6.3 )	46.3 ( +3.7 )	50.6 ( +3.0 )	38.8 ( +4.7 )

Table 6: Retrieval-augmented math problem solving on 200 IMO-level math problems. We show pass@1 averaged over three samples per problem, with absolute improvement over the baseline in parentheses. The discovered Meta-Harness retrieval strategy improves reasoning on these IMO-level problems across all five held-out models, with a 4.7-point average gain over no retriever.

Harness	Auto	Pass (%)
Claude Opus 4.6
Claude Code	×	$58.0$
Terminus 2	×	$62.9$
Mux	×	$66.5$
Droid	×	$69.9$
TongAgents	×	$71.9$

Table 7: Pass rate on TerminalBench-2. Results or others are from the official leaderboard. Meta-Harness ranks #2 among all Opus-4.6 agents and #1 among all Haiku-4.5 agents on this competitive task.

Figure 4: Search-set accuracy over evaluations for all compared text optimizers on online text classification. Each point is one candidate harness; lines track the best-so-far. Per-dataset curves are shown alongside the aggregate. Meta-Harness reaches the final accuracy of OpenEvolve and TTT-Discover within the first 4 evaluations and continues improving, ending more than 10 points above all baselines.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Value
Files read per iteration (median)	82
Files read per iteration (range)	69–99
File type breakdown
Harness source code	41%
Execution traces	40%
Score/summary files	6%

Appendix

Table 8: Proposer file access statistics from the TerminalBench-2 search run (10 iterations, Claude Opus 4.6 ). The proposer reads extensively from the filesystem, with roughly equal attention to prior source code and execution traces.

Panel of Figure 5 — Figure 5: Draft-verification classification harness. The first call produces a draft label from a short retrieved context. The second call retrieves evidence for and against that draft and returns the final prediction.

Panel of Figure 6 — Figure 6: Label-primed query-anchored classification harness. The program builds a single prompt that exposes the label space, then populates it with query-relevant coverage examples and local contrastive pairs.

	Datasets			Avg metrics
Variant	USPTO $\uparrow$	Symptom $\uparrow$	LawBench $\uparrow$	Avg $\uparrow$	Ctx $\downarrow$
Meta-Harness (Draft Verification)	18.0	$85.4$	$17.0$	$40.1$	$5.4$
Meta-Harness (Error-Annotated)	$9.0$	$87.7$	$24.0$	$40.2$	$22.3$
Meta-Harness (CoT Replay)	$13.0$	$88.2$	$25.0$	$42.1$	$23.3$
Meta-Harness (Cluster Coverage)	$12.0$	$86.8$	$33.0$	$43.9$	$31.2$
Meta-Harness (Cascade Retrieval)	$12.0$	$86.8$	$36.0$	$44.9$	$39.2$

Appendix

Table 9: Pareto-optimal discovered variants from the main text-classification search, trading off average accuracy against context cost. The selected system in the main text is Meta-Harness (Label-Primed Query) . Ctx denotes average additional characters in input context (thousands).

Figure 7: Search-set vs. test accuracy per dataset for discovered text-classification strategies. Each pink dot is a discovered strategy; baselines are labeled. The dashed diagonal is $y{=}x$ .

Panel of Figure 8 — Figure 8: Discovered math retrieval harness. A lexical router assigns each query to one of four subject-specific retrieval policies. The selected policy retrieves examples, which are inserted into the final prompt.

Panel of Figure 9 — Figure 9: Discovered TerminalBench-2 harness. The harness inherits Terminus-KIRA’s native tool calling, output cap, and completion checklist (green). The environment bootstrap (red) is the component discovered by Meta-Harness: it gathers a sandbox snapshot before the agent loop begins, eliminating early exploratory turns.

Dataset	Problems	Sol. Len	Proof
OpenMathReasoning	281,743	5,000 †	34%
DeepMath-103K	103,021	5,000 †	0%
NuminaMath-1.5	129,520	1,376	13%
PolyMath	11,083	363	0%
Omni-MATH	4,289	829	0%
FineProofs-SFT	4,275	3,977	100%

Appendix

Table 10: Datasets in the math retrieval corpus (535K problems total). Sol. Len is the median solution length in characters. Proof indicates whether the dataset contains proof-type problems (by answer or problem type field).

Dataset	Problems
IMO-AnswerBench	100
IMO-ProofBench	60
ArXivMath Dec. 2025	17
ArXivMath Jan. 2026	23
Total	200

Appendix

Table 11: Breakdown of the 200-problem IMO-level evaluation set.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 357 papers

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Aug 6, 2026Varun Ursekar, Apaar Shanker, Yash Maurya +4Large Language Model-Guided Optimization LLM Agent Evaluation
Scale AI
MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution
Oct 4, 2026Zhiwei Shang, Yu Huo, Mingrong Gong +6Agent Harness Optimization LLM Agent Workflow Optimization
The Chinese University of Hong Kong, Shenzhen · DeepWisdom · The Chinese University of Hong Kong +3
MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens
Sep 25, 2026Subhojyoti Mukherjee, Md Mehrab TanjimLLM Agent Harnesses Multi-Objective Optimization
Adobe Research

Aug 6, 2026J/K move · Enter open · S save

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser+3

Scale AI

Large Language Model-Guided Optimization LLM Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization AI Agent Benchmarks

Open paper AskPDF arXiv

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Abstract

LLM Agent Harnesses

Explore similar work

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution

MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens