Dynamic Harness Search: Building Multi-Agent Systems Per-Query via Prediction

Som Sagar; Shasha Li; Hejie Cui; Ransalu Senanayake; Sercan Ö. Arık

cs.LGOct 2, 2026

Dynamic Harness Search: Building Multi-Agent Systems Per-Query via Prediction

Authors: Som Sagar, Shasha Li, Hejie Cui, Ransalu Senanayake, Sercan Ö. Arık

Organizations: Google · Arizona State University

Abstract

Agent harnesses specify the roles, instructions, tools, and communication structure used to solve a task, and the right harness depends on the query. Because the value of each design choice is observable only through execution, tailoring a harness to each query has required either executing alternatives at inference time or costly manual design. We introduce SHIFT, which moves execution out of the per-query search loop. A local LLM architect learns a policy over harness-building actions from search, and a value function that predicts, from measured executions, a utility balancing accuracy against execution cost. For each query, Monte Carlo tree search uses these predictions to construct a harness. Across 9,193 tasks in six benchmarks, from math to document and general-assistant tasks, with a Gemini 3.5 Flash executor, SHIFT attains the highest mean accuracy, about 80%, outperforming 17 baselines that span prompting, prompt optimization, and workflow search, and exceeding the strongest baseline by 7.2 percentage points. A cheaper mode of SHIFT also attains a higher mean accuracy than every baseline while using 32% fewer execution tokens than the strongest baseline. We further show that choosing structure, instructions, and tools jointly beats choosing only instructions or only tools by up to 9.1 percentage points, and that learned value selection identifies more accurate harnesses with lower execution cost from candidate pools.

Figures & tables

Figure 1: Tailoring the harness to the query. Three queries ask for a reported revenue, a column total, and a growth rate. (a) A fixed harness runs the same Reader and Solver, with all their tools, on every query: it spends this structure on a simple lookup and still gets both calculations wrong (190 instead of 200, 10% instead of 20%). (b) SHIFT constructs a harness for each query, for example a single reader for the lookup, a reader with a verifier loop for the total, and a reader and a verifier with Python for the growth rate. The architect scores candidates without executing them, and only the constructed harness is run; in this example, all three answers are correct.

Figure 2: Overview of SHIFT. Given a query, the current harness, and its feasible actions, a local Gemma-2-2B architect produces a policy over actions and a utility estimate. MCTS uses these predictions to search over action sequences without executor calls, after which the executor runs only the selected harness. During training, measured outcomes are stored in a replay buffer and used to update the LoRA adapters and policy–value heads.

Figure 3: No single harness fits every task. Utility change (success minus cost) between harnesses with varying structure; small text: accuracy change and token multiplier. More structure can accompany higher accuracy yet lower utility.

Figure 4: One search simulation: (1) select, (2) expand, (3) score with

V_{\theta}

(4) back up. C: Coder, P: Planner, R: Reasoner, K: Critic.

	Short-horizon			Long-horizon, tool-intensive			Aggregate
Method	GSM8K	HotpotQA	MBPP	Sheet	OfficeQA	GAIA	Mean $\,\uparrow$	Cost $\,\downarrow$
Prompting
Direct	97.7 $\pm$ 0.1	67.9 $\pm$ 0.1	93.6 $\pm$ 0.6	0.0	8.9 $\pm$ 3.7	4.6 $\pm$ 1.1	45.5	1.0
CoT	97.4 $\pm$ 0.3	68.4 $\pm$ 0.1	78.3 $\pm$ 0.6	0.0	4.1 $\pm$ 1.4	3.3 $\pm$ 1.1	41.9	0.9 ‡
CoT (few-shot)	97.6 $\pm$ 0.1	68.4 $\pm$ 0.2	82.6 $\pm$ 2.2	0.0	4.9 $\pm$ 2.4	2.6 $\pm$ 1.1	42.7	0.9
Self-Consistency	97.7 $\pm$ 0.1	68.5 $\pm$ 0.0	90.0 $\pm$ 0.4	0.0	6.5 $\pm$ 2.8	6.5 $\pm$ 1.1	44.9	5.2 ‡

Table 1: Accuracy (%, mean and standard deviation over three runs) with a shared Gemini 3.5 Flash executor. The first three benchmarks are short-horizon tasks that a single agent nearly solves; the last three are long-horizon and tool-intensive. Cost is the geometric mean of execution tokens relative to Direct. Bold : best; underline : second best. ‡ Cost average excludes SpreadsheetBench: these methods answer in text, so they score zero there by construction and were not run (Appendix D ).

Mode	GSM8K	Hotpot	MBPP	SpreadSheet.	OfficeQA	GAIA	Mean	Cost
SHIFT-greedy	$98.0_{\pm 0.9}$	$69.2_{\pm 0.1}$	$96.4_{\pm 1.0}$	$87.5_{\pm 2.5}$	$52.8_{\pm 2.8}$	$75.2_{\pm 15.0}$	79.8	120.2
SHIFT-policy-veto	$98.1_{\pm 0.5}$	$69.2_{\pm 0.0}$	$96.4_{\pm 0.6}$	$85.6_{\pm 1.3}$	$59.3_{\pm 9.9}$	$66.0_{\pm 4.9}$	79.1	115.0
SHIFT-value-greedy	$98.0_{\pm 0.1}$	$64.6_{\pm 0.2}$	$95.6_{\pm 0.6}$	$58.6_{\pm 1.3}$	$10.6_{\pm 3.7}$	$34.0_{\pm 3.0}$	60.2	5.3
SHIFT-search	$98.2_{\pm 0.5}$	$68.7_{\pm 0.1}$	$96.6_{\pm 0.2}$	$87.8_{\pm 1.7}$	$59.3_{\pm 5.1}$	$68.6_{\pm 5.2}$	79.9	84.3
SHIFT-value	$97.7_{\pm 0.1}$	$67.5_{\pm 0.2}$	$96.1_{\pm 0.8}$	$81.7_{\pm 0.8}$	$39.0_{\pm 6.5}$	$64.7_{\pm 7.1}$	74.5	27.7

Table 2: Decoding comparison using the same trained architect within each benchmark. Cells report accuracy (%, mean

\pm

sample standard deviation over three evaluation runs). Mean weights all six benchmarks equally; Cost is the geometric mean of execution-token ratios to Direct.

Figure 7: Solved test queries of SHIFT-value: execution tokens, agents in the harness (shade), and tool calls (size).

Figure 10: HotpotQA accuracy when choosing from

k

candidates.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Actions in the vocabulary	40
Top actions per node, $K_{\mathrm{exp}}$ (stop always retained)	10
Maximum number of actions, $H$	10
Search simulations, $B$	64
Exploration constant, $c_{\mathrm{puct}}$	2.5
Policy temperature, $T_{\pi}$	1

Appendix

Table 5: Search settings shared by all benchmarks.

ID	Target	Action
1–6	Graph	Add a Planner, Researcher, Reasoner, Critic, Verifier, or Aggregator agent.
7–8	Graph	Add a feedback loop from the Verifier or the Critic back to the executing agent.
9	Graph	Give every executing agent the full tool set.
10	Graph	Stop and return the current harness.
11	Coder	Think step-by-step before generating code.
12	Coder	Write clean, concise code without extra comments.

Appendix

Table 6: The action vocabulary. Directives are appended at most once to their target agent.

Benchmark	Tokens $\rho_{t}$	Latency $\rho_{\ell}$ (s)
GSM8K	30,000	25
HotpotQA	40,000	30
MBPP	60,000	60
SpreadsheetBench	250,000	100
OfficeQA	1,500,000	400
GAIA	300,000	200

Appendix

Table 7: Benchmark-specific reference scales for the token and latency penalties (Equation 7 ).

Setting	Value
LoRA rank / scale / dropout	16 / 32 / 0.05
LoRA target layers	attention query, key, value, output
Optimizer	AdamW, learning rate $10^{-4}$ , weight decay 0.01
Learning-rate schedule	cosine, minimum $10^{-5}$
Gradient clipping	1.0
Epochs	4

Appendix

Table 8: Training hyperparameters for the Gemma 4 E2B architect.

Figure 11: How harness composition changes over training. Each cell gives the share (%) of final-epoch training harnesses that contain a role (Verifier and Aggregator omitted), with the change since the first epoch below. Gemini 3.5 Flash executor.

Figure 12: Mean policy loss over four training epochs, one run per benchmark, averaging records with nonzero total loss. Hotpot, Sheet, and Office denote HotpotQA, SpreadsheetBench, and OfficeQA.

Figure 13: Execution-token savings relative to greedy construction, paired by question. Left: reduction in mean execution tokens, with 95% bootstrap intervals over questions. Right: share of questions on which each strategy uses fewer tokens than greedy construction.

Figure 14: Cost and harness size per test query (all test runs). (a) Distribution of execution tokens per query for SHIFT-search and SHIFT-value. (b) Harnesses chosen by SHIFT-value: bubble area is the share of test queries whose harness has that many agents (labels for shares of at least 25%), and color is the median execution tokens of those queries.

Figure 15: Architect construction cost on one H100 NVL GPU (median over ten questions, three repeats each). (a) Time to construct one harness at the default budget of 64 simulations; whiskers show the interquartile range. (b) Time per simulation as the budget grows. Benchmarks marked ∗ (dashed in b) use the HotpotQA architect with the benchmark’s tools.

Benchmark	Task	Success criterion	Train	Val	Test
GSM8K	Grade-school math word problems	Final number equals the reference	472	128	1,319
HotpotQA	Multi-hop questions over supplied passages (distractor setting)	Exact match after standard answer normalization	472	128	7,405
MBPP	Short Python functions from a description (sanitized set)	Generated code passes all unit tests	120	43	257
SpreadsheetBench	Spreadsheet manipulation from natural-language instructions	Output workbook matches the reference on every test case	220	60	120
OfficeQA Pro	Questions over a corpus of 697 U.S. Treasury Bulletins	Answer matches the reference within 1%	73	19	41
GAIA	General assistant tasks needing web search, files, and reasoning	Normalized answer matches the reference	90	24	51

Appendix

Table 9: Benchmarks, success criteria, and data splits (number of tasks). Training, validation, and test sets are disjoint.

	Agents		Tools		Directives
Benchmark	1	$\geq 4$	none	all	0	$\geq 2$
GSM8K	98.2 (114)	98.4 (835)	98.4 (127)	97.8 (965)	98.0 (665)	97.9 (419)
MBPP	94.1 (34)	99.0 (210)	95.6 (68)	99.4 ∗ (319)	98.0 (247)	100.0 (37)
HotpotQA	75.6 (90)	84.3 ∗ (1,116)	72.4 (76)	83.7 ∗ (1,611)	79.6 (554)	83.1 (278)
SpreadsheetBench	55.3 (38)	82.8 ∗ (476)	44.8 (29)	87.3 ∗ (700)	65.4 (208)	83.0 ∗ (406)
GAIA	40.0 (25)	73.9 ∗ (134)	35.3 (17)	66.9 ∗ (290)	50.0 (90)	72.2 ∗ (158)

Appendix

Table 10: Success rates (%) of executions with the least and most of each structure type; the number of executions is in parentheses. ∗ The two groups differ at

p<0.05

(two-sided Fisher exact test).

Figure 16: Question-level comparison with Trace, the strongest baseline, on OfficeQA (41 questions) and GAIA (51). Each cell counts the questions that SHIFT-search (rows) and Trace (columns) solve in the given number of their three runs; cells above the diagonal are questions SHIFT-search solves more often.

Actions allowed	Accuracy (%)	Change vs. full (%)
All actions	$69.4_{\pm 0.5}$	–
Tool grants only	$67.9_{\pm 0.3}$	$-1.5$ [ $-3.1$ , $-0.1$ ]
Directives only	$60.3_{\pm 0.9}$	$-9.1$ [ $-12.0$ , $-6.4$ ]

Appendix

Table 11: HotpotQA accuracy under restricted action spaces (500 questions, mean

\pm

SD over three runs; change with paired 95% bootstrap interval).

Figure 17: Choosing among the same candidate harnesses with the value head versus uniformly at random (400 HotpotQA and 200 GSM8K questions, Gemma 3 27B executor). (a) Accuracy. (b) Execution tokens of the selected harness.

Figure 18: MATH-500 accuracy by difficulty level for SHIFT, using the GSM8K architect without retraining.

Configuration	Accuracy (%)	Executor tokens / problem
SHIFT (learned policy, learned value)	83.6	8,049
Learned policy, constant value	82.4	7,614
Uniform policy, constant value	82.0	6,283
Uniform policy, learned value	81.2	5,585
Fixed single Coder	81.2	1,064

Appendix

Table 12: MATH-500 accuracy with Llama 4 Scout, using the GSM8K architect without retraining (500 problems per row, one execution each). Uniform policy: feasible actions receive equal prior probability. Constant value: every harness receives a predicted utility of 0.5.

Agent Harness Optimization

Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.

See all 185 papers

Oct 4, 2026

MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution

An agent harness is the code that organizes context, maintains state, and coordinates tool calls for a language model. We study how to improve the harness under a limited evaluation budget while keeping model weights fixed. Our method, MESH-Harness, organizes each harness into functional modules with explicit role-specific interfaces, allowing alternative implementations of each module to be substituted and recombined. It uses shared module representations and full-covariance LinUCB to score candidate combinations based on predicted performance and exploration value. Mixed-start coordinate ascent selects complete configurations for evaluation without enumerating the combinatorial space. Validation traces then guide local code edits, and the resulting candidates are incorporated into fixed-capacity role-specific pools for subsequent recombination. On text tasks, retrieval-augmented mathematical reasoning, code generation, and interactive scientific tasks, MESH-Harness outperforms Meta-Harness by 5.70, 7.01, 2.00, and 5.00 points, respectively, under matched candidate-evaluation budgets. Iterative harness optimization improves MESH-Harness by 5.63-7.79 points over its first-round configurations. For the reported configurations, aggregate test-time cost is 44.2% lower than that of Meta-Harness, while total cost including search is 14.6% lower. These results show that combining module-level design reuse with feedback-driven compositional search can systematically improve agent harnesses while keeping overall optimization cost under control.

Zhiwei Shang, Yu Huo, Mingrong Gong +6

The Chinese University of Hong Kong, Shenzhen · DeepWisdom · The Chinese University of Hong Kong +3

Agent Harness Optimization LLM Agent Workflow Optimization LLM Agent Self-Improvement LLM Agent Harnesses

Ask PDF

Aug 6, 2026

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Varun Ursekar, Apaar Shanker, Yash Maurya +4

Scale AI

Large Language Model-Guided Optimization LLM Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization AI Agent Benchmarks

Ask PDF

Sep 1, 2026

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16

LLM Agent Evaluation Agent Harness Optimization AI Agent Benchmarks

Ask PDF

Abstract

Agent Harness Optimization

Explore similar work

MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?