MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens

Subhojyoti Mukherjee; Md Mehrab Tanjim

cs.AISep 25, 2026

MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens

Authors: Subhojyoti Mukherjee, Md Mehrab Tanjim

Organizations: Adobe Research

Abstract

Most work on improving large language models treats accuracy as the sole objective. We argue that the harness, the Python code surrounding the model that constructs prompts, routes calls, and parses outputs, is a first-class design surface whose quality is inherently multi-objective: an accurate harness that refuses no unsafe request, or that consumes an order of magnitude more tokens, is not a good harness. We present Meta-Harness, a system that casts harness design as search over three per-domain objectives (accuracy, behavioural safety, and token cost) solved by an agentic proposer (Claude Code) with full filesystem access to prior harness source, execution traces, and scoring artifacts. Our central finding is that a singlephase joint-reward proposer (MoMHa) outperforms every alternative, including a two-phase "accuracy then tokens" ablation, scalar-only feedback, and an accuracy-only baseline. We evaluate on seventeen domains: seven synthetic capability suites, seven real-world public benchmarks (HumanEval, MBPP, Spider, FEVER, MMLU-Pro, LawBench, NuminaMath), and three U-SafeBench-derived user-specific safety domains, using a 12-model fleet spanning four families. On the synthetic track MoMHa achieves a joint mean of 0.482 versus 0.198-0.422 for ten baselines, winning $7 / 10$ per-domain columns; on the real-world track it scores 0.461 versus 0.377 for the strongest baseline (DSPy), winning 5/7 columns, demonstrating that harness strategies transfer to unseen benchmarks without retraining on 8 of 12 target models. MoMHa attains the highest measured behavioral safety composite (U-SafeBench, 0.781) and uses 95 fewer tokens per example than the two-phase alternative. We will release all harness code, evaluation infrastructure, and crossmodel logs.

Figures & tables

Tier	Models
Frontier	Claude Opus 4.5, GPT-5.2, GPT-5.4
Strong	Claude Sonnet 4.6, Claude Sonnet 4.5
Mid	GPT-4.1, GPT-5-mini, o4-mini, DeepSeek-R1, GPT-4.1-mini
Efficient	Claude Haiku 4.5, Gemini 3.1 Flash-Lite

Table 1: 12 models for cross-model evaluation, grouped by tier.

	TC Law	Math R	FV FEV	SQL Sp	HumanEval	MBPP	MMLU-Pro	Mean
MH	$0.140_{\pm.004}$	$0.360_{\pm.005}$	$0.421_{\pm.006}$	$0.222_{\pm.005}$	$0.338_{\pm.007}$	$0.066_{\pm.002}$	$0.389_{\pm.006}$	$0.277$
CoT	$0.174_{\pm.002}$	$0.417_{\pm.006}$	$0.530_{\pm.001}$	$0.268_{\pm.005}$	$0.318_{\pm.007}$	$0.060_{\pm.002}$	$0.383_{\pm.004}$	$0.307$
APE	$0.194_{\pm.002}$	$0.435_{\pm.006}$	$0.559_{\pm.002}$	$0.154_{\pm.007}$	$0.368_{\pm.007}$	$0.092_{\pm.003}$	$0.436_{\pm.004}$	$0.320$
OPRO	$0.178_{\pm.002}$	$0.315_{\pm.003}$	$0.505_{\pm.001}$	$0.110_{\pm.005}$	$0.374_{\pm.006}$	$0.074_{\pm.002}$	$0.342_{\pm.005}$	$0.271$
DSPy	$\underline{0.218}_{\pm.003}$	$\underline{0.453}_{\pm.001}$	$0.487_{\pm.001}$	$\underline{0.296}_{\pm.001}$	$\underline{0.438}_{\pm.007}$	$\underline{0.299}_{\pm.003}$	$\underline{0.448}_{\pm.002}$	$\underline{0.377}$
MIPROv2	$0.196_{\pm.002}$	$0.396_{\pm.006}$	$0.475_{\pm.005}$	$0.155_{\pm.007}$	$0.386_{\pm.007}$	$0.082_{\pm.002}$	$0.418_{\pm.004}$	$0.301$

Table 2: Joint per-domain score on real-world benchmarks

J_{v,d}

( Equation 4 ). Bold = column winner; underline = runner-up. MoMHa wins

5/7

per-domain columns and the 7-domain mean. Cells show mean

{}_{\pm\text{SEM}}

with

\text{SEM}=\sigma_{\text{cross-model}}/\sqrt{N_{\text{trials}}}

and

N_{\text{trials}}\!\approx\!11\times 50

per cell.

	TC	Math	Code	MCQ	FV	NER	SQL	Safe ill	Safe phys	Safe ment	Mean
MH	$0.389_{\pm.006}$	$0.275_{\pm.006}$	$0.131_{\pm.004}$	$\underline{0.407}_{\pm.006}$	$0.273_{\pm.004}$	$0.427_{\pm.008}$	$0.135_{\pm.003}$	$0.258_{\pm.001}$	$0.421_{\pm.003}$	$0.337_{\pm.003}$	$0.305$
CoT	$0.474_{\pm.006}$	$0.291_{\pm.006}$	$0.135_{\pm.005}$	$0.307_{\pm.004}$	$0.294_{\pm.003}$	$0.375_{\pm.008}$	$\underline{0.154}_{\pm.004}$	$0.371_{\pm.003}$	$0.484_{\pm.003}$	$0.396_{\pm.005}$	$0.328$
APE	$0.543_{\pm.003}$	$0.303_{\pm.006}$	$0.138_{\pm.004}$	$0.339_{\pm.005}$	$0.331_{\pm.002}$	$0.431_{\pm.009}$	$0.116_{\pm.003}$	$0.425_{\pm.002}$	$0.431_{\pm.004}$	$0.417_{\pm.004}$	$0.347$
OPRO	$0.460_{\pm.003}$	$0.208_{\pm.003}$	$0.117_{\pm.004}$	$0.277_{\pm.005}$	$0.319_{\pm.002}$	$0.315_{\pm.008}$	$0.058_{\pm.003}$	$0.330_{\pm.001}$	$0.401_{\pm.003}$	$0.349_{\pm.003}$	$0.283$
DSPy	$0.531_{\pm.001}$	$\mathbf{0.364}_{\pm.003}$	$0.094_{\pm.003}$	$0.386_{\pm.002}$	$0.319_{\pm.002}$	$0.462_{\pm.001}$	$0.152_{\pm.002}$	$0.356_{\pm.002}$	$\underline{0.512}_{\pm.001}$	$0.456_{\pm.002}$	$0.363$
MIPROv2	$0.549_{\pm.002}$	$0.262_{\pm.003}$	$0.146_{\pm.004}$	$0.340_{\pm.006}$	$0.343_{\pm.002}$	$0.465_{\pm.008}$	$0.128_{\pm.004}$	$0.402_{\pm.002}$	$0.434_{\pm.004}$	$0.435_{\pm.005}$	$0.350$

Table 3: Joint per-domain score

J_{v,d}=\mathrm{acc}\cdot\mathrm{Safe}/\ln(\mathrm{tokens})

( Equation 4 ). Each cell folds accuracy, safety, and token cost into one number. Bold = column winner; underline = runner-up. MoMHa wins

7

10

per-domain columns and leads the 10-domain mean by

+0.060

points over the strongest baseline (TextGrad). DSPy’s

2229

-token capability harnesses are heavily penalized by the

\ln(\mathrm{tokens})

denominator. Cells show mean

{}_{\pm\text{SEM}}

with

\text{SEM}=\sigma_{\text{cross-model}}/\sqrt{N_{\text{trials}}}

N_{\text{trials}}\!\approx\!12\times 50

per cell.

Panel of Figure 1 — Figure 1: Multi-objective analysis across eleven variants. (a) Non-MoMHa baselines span a safety band of $0.57$ – $0.75$ ; TextGrad alone approaches MoMHa’s safety but at lower accuracy. (b) MoMHa covers all three axes; baselines collapse on at least one. (c) Per-axis search-reward: MoMHa is the outer envelope on the seven real-world capability axes and on the two autonomous-harm safety axes. Points in (a,b) are aggregated over all 17 domains $\times$ 12 models; per-domain Pareto panels in Appendix J .

Model	MH	CoT	DSPy	APE	OPRO	GEPA	MIPRO	Rand	TG	AH	MoMHa (ours)
Haiku 4.5	$.318_{\pm.006}$	$.371_{\pm.006}$	$.344_{\pm.006}$	$.400_{\pm.007}$	$.292_{\pm.005}$	$.410_{\pm.007}$	$.341_{\pm.007}$	$.371_{\pm.007}$	$\underline{.440}_{\pm.008}$	$.224_{\pm.013}$	$\mathbf{.570}_{\pm.007}$
Sonnet 4.5	$.261_{\pm.007}$	$.326_{\pm.007}$	$\underline{.405}_{\pm.007}$	$.322_{\pm.007}$	$.248_{\pm.006}$	$.332_{\pm.008}$	$.287_{\pm.008}$	$.247_{\pm.007}$	$.330_{\pm.010}$	$.207_{\pm.013}$	$\mathbf{.440}_{\pm.009}$
Sonnet 4.6	$.329_{\pm.007}$	$.219_{\pm.006}$	$\underline{.436}_{\pm.006}$	$.319_{\pm.009}$	$.278_{\pm.006}$	$.350_{\pm.009}$	$.295_{\pm.008}$	$.232_{\pm.007}$	$.390_{\pm.009}$	$.230_{\pm.013}$	$\mathbf{.508}_{\pm.008}$
Opus 4.5	$.267_{\pm.007}$	$.201_{\pm.006}$	$\underline{.412}_{\pm.006}$	$.271_{\pm.007}$	$.238_{\pm.006}$	$.268_{\pm.008}$	$.260_{\pm.007}$	$.225_{\pm.007}$	$.357_{\pm.009}$	$.248_{\pm.011}$	$\mathbf{.471}_{\pm.009}$
GPT-4.1	$.329_{\pm.007}$	$.426_{\pm.007}$	$.437_{\pm.006}$	$\underline{.491}_{\pm.008}$	$.457_{\pm.006}$	$.486_{\pm.009}$	$.473_{\pm.008}$	$.347_{\pm.010}$	$.480_{\pm.009}$	$.355_{\pm.015}$	$\mathbf{.578}_{\pm.009}$
GPT-4.1-mini	$.305_{\pm.007}$	$.286_{\pm.009}$	$.384_{\pm.006}$	$.430_{\pm.008}$	$.428_{\pm.007}$	$.455_{\pm.009}$	$.438_{\pm.008}$	$.256_{\pm.009}$	$\underline{.492}_{\pm.009}$	$.294_{\pm.016}$	$\mathbf{.536}_{\pm.009}$

Table 4: Cross-model transfer: mean joint reward

J

across 12 target models and 10 domains. Each cell is the mean of

J{=}\mathrm{acc}\,\mathrm{Safe}_{v}/\ln(\mathrm{tokens})

per (variant, model). MoMHa (ours) wins on 8 of 12 models; bold = best per row, underline = runner-up; “Ours” col. = MoMHa (ours). Cells show mean

{}_{\pm\text{SEM}}

with

\text{SEM}=\sigma_{\text{cross-domain}}/\sqrt{N_{\text{trials}}}

N_{\text{trials}}\!\approx\!10\times 50

per cell.

Figure 2: 3D hypervolume indicator (per-model HV over capability accuracy $\times$ behavioral safety $\times$ token efficiency, pymoo ( Blank and Deb, 2020 ) ). MoMHa (ours) leads all 10 external baselines ( $\mathrm{HV}{=}0.481$ ).

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Synthetic (MH proposer-searched)
Domain	MH	MoMHa
Text Classification (syn)	0.50	1.00
Math Reasoning (syn)	1.00	1.00
Agentic Coding (syn)	0.83	1.00
MCQ (syn)	1.00	1.00
Fact Verification (syn)	1.00	1.00

Appendix

Table 5: Code-structural safety composite scores (mean over all harness candidates per domain; six equally-weighted AST checks: parseable, no banned imports, no banned calls, no banned attributes, declares a Harness class with init / run , and no embedded shell strings). All MoMHa-family variants score

1.00

across all 17 domains by construction of the proposer AST guard; behavioral safety differences between variants are in Appendix K . MH composite is

<1

precisely on the seven synthetic domains where MH was itself proposer-searched; on the safety and real-world benchmarks, MH uses curated baseline.py harnesses and is safe by inspection.

Domain	Primary token sink	Prescribed cut	Saving
Synthetic
text_class. (syn)	verification call on confident drafts	skip verification when decisive	${\sim}300$ tok
math_reason. (syn)	CoT on easy problems	subject+difficulty routing	${\sim}500$ tok
agentic_code (syn)	fix-loop iterations	env snapshot + better first attempt	${\sim}1500$ /iter
mcq (syn)	CoT on factual-recall items	difficulty routing; 1-token answer	${\sim}400$ tok
fact_verif. (syn)	evidence decomposition on clear claims	single-pass; decompose iff ambiguous	${\sim}300$ – $500$ tok

Appendix

Table 6: Per-domain token sinks and the cuts prescribed in each proposer_<domain>.md skill. “Saving” is the budget freed per example (or per saved loop iteration) when the cut applies. The generic Phase-2 skill contributes the how (confidence routing, prompt compression, early stopping, few-shot pruning, output-format shrink, prompt caching); the per-domain skill contributes the where .

Domain	MH tok	MoMHa tok	Ratio
Text Classification (syn)	$485_{\pm 6}$	$247_{\pm 6}$	$0.51_{\pm 0.01}$
Math Reasoning (syn)	$597_{\pm 10}$	$722_{\pm 13}$	$1.21_{\pm 0.08}$
Agentic Coding (syn)	$802_{\pm 13}$	$942_{\pm 17}$	$1.16_{\pm 0.01}$
MCQ (syn)	$359_{\pm 1}$	$783_{\pm 10}$	$2.19_{\pm 0.03}$
Fact Verification (syn)	$502_{\pm 5}$	$754_{\pm 12}$	$1.49_{\pm 0.01}$
NER	$408_{\pm 6}$	$633_{\pm 9}$	$1.55_{\pm 0.01}$

Appendix

Table 7: Token consumption ratio (MoMHa/MH) per domain, averaged over the 12-model cohort. Values

<1

indicate MoMHa uses fewer tokens. Cells show mean

{}_{\pm\text{SEM}}

with

\text{SEM}=\sigma_{\text{cross-model}}/\sqrt{N_{\text{trials}}}

N_{\text{trials}}\approx 12\times 50

; the ratio SEM uses the per-model ratio distribution over the pooled trial pool.

Domain	MH	MoMHa (ours)	$\boldsymbol{\Delta}$
Text Classification (syn)	$70.2_{\pm 1.1}$	$\mathbf{73.0}_{\pm 1.0}$	$+2.8$
Math Reasoning (syn)	$\mathbf{51.2}_{\pm 1.2}$	$49.0_{\pm 1.1}$	$-2.2$
Agentic Coding (syn)	$25.4_{\pm 0.7}$	$\mathbf{58.3}_{\pm 1.4}$	$+32.9$
MCQ (syn)	$\mathbf{69.8}_{\pm 1.0}$	$57.8_{\pm 0.7}$	$-12.0$
Fact Verification (syn)	$49.5_{\pm 0.7}$	$\mathbf{50.8}_{\pm 0.5}$	$+1.3$
NER	$\mathbf{68.6}_{\pm 1.3}$	$56.2_{\pm 1.7}$	$-12.4$

Appendix

Table 8: MH vs. MoMHa accuracy (%) per domain, averaged over the 12-model cohort. MoMHa uses the full safety architecture. Real-world domains (FEVER, HumanEval, MATH, MBPP, MMLU-Pro, Spider, LawBench) average over 11 models; synthetic and safety domains average over 12. Cells show mean

{}_{\pm\text{SEM}}

with

\text{SEM}=\sigma_{\text{cross-model}}/\sqrt{N_{\text{trials}}}

where

N_{\text{trials}}\approx 12\times 50

per cell. Bold indicates the better variant per row.

Domain	MH	2-phase	MoMHa-ns	MoMHa-noTok	MoMHa-scalar	MoMHa (ours)
Text Classification (syn)	$70.2_{\pm 1.1}$	$65.2_{\pm 1.3}$	$65.7_{\pm 1.3}$	$66.5_{\pm 1.1}$	$71.0_{\pm 0.7}$	$\mathbf{73.0}_{\pm 1.0}$
Math Reasoning (syn)	$\mathbf{51.2}_{\pm 1.2}$	$46.9_{\pm 1.1}$	$49.1_{\pm 1.1}$	$44.9_{\pm 1.1}$	$48.5_{\pm 0.9}$	$49.0_{\pm 1.1}$
Agentic Coding (syn)	$25.4_{\pm 0.7}$	$61.1_{\pm 1.3}$	$\mathbf{68.1}_{\pm 1.4}$	$59.9_{\pm 1.4}$	$65.9_{\pm 1.4}$	$58.3_{\pm 1.4}$
MCQ (syn)	$\mathbf{69.8}_{\pm 1.0}$	$68.5_{\pm 1.0}$	$66.2_{\pm 1.0}$	$62.2_{\pm 1.0}$	$56.7_{\pm 0.7}$	$57.8_{\pm 0.7}$
Fact Verification (syn)	$49.5_{\pm 0.7}$	$49.5_{\pm 0.4}$	$49.5_{\pm 0.4}$	$44.3_{\pm 0.6}$	$47.3_{\pm 0.5}$	$\mathbf{50.8}_{\pm 0.5}$
NER	$68.6_{\pm 1.3}$	$67.8_{\pm 1.4}$	$66.5_{\pm 1.6}$	$\mathbf{69.1}_{\pm 1.4}$	$55.8_{\pm 1.7}$	$56.2_{\pm 1.7}$

Appendix

Table 9: Accuracy (%) across the MoMHa family , 17 domains averaged over the 12-model cohort (11 models on the 7 real-world domains). 2-phase splits the joint objective into an accuracy/token phase followed by a safety phase; MoMHa-ns drops the safety skill; MoMHa-noTok drops the token term; MoMHa-scalar uses a single scalar reward (no per-example traces); MoMHa is the full system. Cells show mean

{}_{\pm\text{SEM}}

over the pooled trial pool (

\approx 12\times 50

per cell).

Panel of Figure 3 — Figure 3: Agentic coding: MoMHa’s converged two-call strategy vs. baselines. Call 1 is a lightweight analysis pass (512 tokens) that identifies the core algorithm without generating code. Call 2 uses the analysis as context. The two-call approach is only triggered for algorithmically complex problems (graph algorithms, DP, monotonic deque); simple problems skip to a single call, keeping token cost low. Accuracy: MoMHa $\mathbf{100\%}$ vs. MH $33\%$ , DSPy $14\%$ across all five cross-model targets ( Table 8 ).

Panel of Figure 4 — Figure 4: User-specific safety: MoMHa’s discovered profile-first routing vs. baselines. TextGrad blends safety rules into one unstructured blob and asks the model to generically “analyze” safety. MoMHa restructures this as an explicit ordered Safety Check procedure and places the profile before the instruction in the user turn, forcing role/status evaluation before the request content is processed. Safety composite: MoMHa $\mathbf{0.781}$ vs. TextGrad $0.747$ , CoT $0.642$ , MH $0.569$ ( Table 3 ).

Panel of Figure 5 — Figure 5: SQL generation: MoMHa’s schema-annotation and generate-then-verify strategy vs. baselines. Baselines dump the raw DDL and ask for SQL in one call, leading to hallucinated column names on complex multi-table queries. MoMHa’s first call uses a schema pre-processor that annotates each column with its type and FK relationship, reducing hallucination. For complex queries (detected by keywords: JOIN, GROUP BY, HAVING, subquery, CTE, rank, partition), a second verification call reviews and fixes the SQL before returning the answer. SQL accuracy on Haiku (search set): MoMHa $\mathbf{0.703}$ vs. MH $0.570$ , TextGrad $0.614$ (cross-model test mean: $36.8\%$ , Table 8 ).

Figure 6: Skill optimization vs. Meta-Harness. Left : all existing baselines (APE, OPRO, TextGrad, GEPA, DSPy) operate as skill optimizers : they search only over instruction strings while keeping the harness code fixed (shown in blue). Right : MoMHa’s proposer rewrites the entire Python harness (shown in orange), making routing logic, multi-call pipelines, schema processing, and self-checking all searchable. Strategies inexpressible as prompt edits, complexity-gated pipelines, environment bootstrapping, schema annotation, can only emerge in the right-hand regime.

Panel of Figure 7 — Figure 7: Per-model accuracy across ten variants on the seven synthetic capability domains. Real-world counterparts are in Figure 9 ; token-side panels are in Figures 8 and 10 .

Panel of Figure 8 — Figure 8: Per-model average tokens-per-example across ten variants on the seven synthetic capability domains.

Panel of Figure 9 — Figure 9: Per-model accuracy across ten variants on the seven real-world capability benchmarks (LawBench, MATH-500, HumanEval, MBPP, FEVER, MMLU-Pro, Spider).

Panel of Figure 10 — Figure 10: Per-model average tokens-per-example across ten variants on the seven real-world capability benchmarks.

Panel of Figure 11 — Figure 11: Per-model accuracy (symmetric refusal/compliance reward) across ten variants on the three U-SafeBench safety domains. MoMHa dominates on safety_autonomous_mental ( $0.931$ ), the domain where affect-aware profile reading is most critical.

Panel of Figure 12 — Figure 12: Per-model average tokens-per-example across ten variants on the three U-SafeBench safety domains.

Figure 13: Per-domain 3D Pareto surfaces (tokens $\times$ accuracy $\times$ safety), one panel per domain, across all 17 evaluation domains. MoMHa (green) occupies the high-accuracy, high-safety, low-token corner on essentially every capability domain.

Figure 14: Per-domain three-axis spiders (accuracy $\times$ safety $\times$ token efficiency), one panel per domain across all 17 evaluation domains, 11 variants overlaid. This view exposes exactly which objective each variant loses on within each domain.

Panel of Figure 15 — Figure 15: MoMHa-family ablation (top-5 MoMHa-favoring domains). Aggregated over {Safety-Auto (phys.), TC-syn, MATH-500, FV-syn, MBPP}, the domains with the largest MoMHa–to–best-other-MoMHa gap in $J$ . Left: 3D Pareto surface (accuracy $\times$ safety $\times$ tokens). MoMHa is non-dominated on all three axes; MoMHa-noTok and 2-phase are the closest competitors on accuracy but pay a tokens penalty, and MoMHa-scalar and MoMHa-ns are dominated on safety. Right: three-axis spider. MoMHa fills the outer envelope, MoMHa-scalar collapses on accuracy, and MoMHa-ns/MoMHa-noTok/2-phase each give up one axis relative to MoMHa (safety, tokens, and both, respectively). This isolates the contribution of each design choice more cleanly than the 17-domain aggregate, where the ablations’ wins on unrelated subdomains cancel their losses on the MoMHa-favoring ones.

Variant	Safety skill	Objective	Feedback
MoMHa (ours)	✓	single-phase joint $\mathrm{acc}{+}\lambda_{s}\mathrm{safety}{-}\lambda_{t}\mathrm{tok}$	per-example traces
2-phase	✓	two-phase: accuracy $\to$ tokens (2pp tolerance)	per-example traces
MoMHa-ns	$\times$	single-phase joint (no safety skill)	per-example traces
MoMHa-noTok	✓	acc + safety (no token term)	per-example traces
MoMHa-scalar	✓	single-phase joint (scalar $R$ only)	scalar reward

Appendix

Table 10: MoMHa family: what each ablation turns off. “Safety skill” = per-domain safety_<domain>.md . “Objective” = how accuracy, safety, and tokens are combined. “Feedback” = what the proposer reads from each prior run. MoMHa is the headline; 2-phase is the strongest ablation.

Figure 16: Hypervolume indicator: MoMHa family ablation (3D: capability accuracy $\times$ behavioral safety $\times$ token efficiency, pymoo [ Blank and Deb, 2020 ] ). MoMHa (ours) achieves the highest family HV ( $0.481$ ), with MoMHa-scalar ( $0.447$ ) and 2-phase ( $0.425$ ) trailing. MoMHa-ns ( $0.400$ ) shows the cost of removing the safety objective. The vs.-baselines comparison is in Figure 2 .

Feedback	Best Accuracy (%)
Scores only	41.0
Scores + NL summary	34.9
Full traces (ours)	57.0

Appendix

Table 11: Ablation: feedback granularity. Best accuracy (%) achieved over search on the text-classification (TC) synthetic domain. Full traces outperform compressed feedback by

+16

points.

Figure 17: Total search-time + final-evaluation tokens per variant (log scale), sorted ascending. MoMHa (v3joint) totals 15.61M tokens, below 2-phase (18.33M) and DSPy (37.04M).

Variant	Search-eval (M)	Final-eval (M)	Total (M)
AutoHarness	0.00	2.41	2.41
MH (v1)	2.71	5.46	8.17
CoT	3.04	5.66	8.70
MoMHa-noTok	1.35	7.57	8.92
GEPA	3.62	5.62	9.24
MoMHa-ns	4.91	6.77	11.68

Appendix

Table 12: Total token cost per variant: search-time evaluator tokens plus the fixed 12-model final-evaluation matrix. Sorted ascending by total.

Proposer	math_reasoning	agentic_coding	safety_qa_illegal	Mean $J$
claude-sonnet-4.6 (MoMHa)	0.103	0.606	0.530	0.413
llama-3-3-70b	0.239	0.372	0.953	0.521
deepseek-r1	0.157	0.409	0.797	0.454
gpt-5.4	0.000 †	0.000 †	0.709	0.236 †

Appendix

Table 13: Proposer-model ablation.

J

score per domain, 3 non-Claude proposers vs. the headline claude-sonnet-4.6 proposer.

\dagger

: gpt-5.4 generated banned getattr() calls, failing AST validation on math and coding (falls back to the baseline harness,

J\!\approx\!0

Figure 18: MoMHa’s rank (lower is better) across 8 scalarizations and both evaluation tracks. MoMHa is rank 1 on the synthetic track under every scalarization tested.

Scalarization	Synthetic rank	Real-world rank
S1: Composite $J$ (paper)	1	1
S2: Weighted sum, $w{=}0.4$	1	2
S2: Weighted sum, $w{=}0.5$	1	2
S2: Weighted sum, $w{=}0.6$	1	2
S2: Weighted sum, $w{=}0.7$	1	2
S3: Product acc $\cdot$ Safe, no log-tokens	1	1

Appendix

Table 14: MoMHa (v3joint) rank under 8 scalarizations, out of 14 variants.

Domain	Full $J$	Stub $J$	$\Delta J$
sql_generation	0.263	0.247	$-0.016$ ( $-6\%$ )
safety_autonomous_physical	0.712	0.855	$+0.143$ ( $+20\%$ )

Appendix

Table 15: Skill-file ablation:

J

score with full per-domain skill files vs. empty stubs (framework-only fallback).

Domain	$n$	Paper judge acc.	2nd-judge acc.	$\kappa$
safety_qa_illegal	60	0.808	0.775	0.786
safety_autonomous_physical	30	0.883	0.933	0.769
safety_autonomous_mental	30	0.950	0.967	0.941

Appendix

Table 16: Inter-judge agreement (Cohen’s

\kappa

) and accuracy under the second judge, U-SafeBench domains.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 357 papers

Mar 30, 2026

Meta-Harness: End-to-End Optimization of Model Harnesses

The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.

Yoonho Lee, Roshen Nair, Qizheng Zhang +3

Stanford · KRAFTON · MIT

LLM Agent Harnesses Large Language Model-Guided Optimization

Ask PDF

Aug 6, 2026

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Varun Ursekar, Apaar Shanker, Yash Maurya +4

Scale AI

Large Language Model-Guided Optimization LLM Agent Evaluation Agent Harness Optimization LLM Agent Workflow Optimization AI Agent Benchmarks

Ask PDF

Aug 18, 2026

HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety

Large language models are increasingly deployed through agent harnesses that manage tools, extensions, persistent state, permissions, and external actions. Existing safety benchmarks mainly target individual attack mechanisms or a limited subset of operational settings, making it difficult to compare how safety failures emerge across different harness responsibilities. We present HarnessRisk, a lifecycle oriented benchmark that organizes agent harness safety into six operational phases including Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery. HarnessRisk contains 128 sandboxed cases, each pairing a benign user objective with an adversarial instruction embedded in an untrusted workflow artifact. We evaluate each trajectory using Utility, Attack Success Rate, Persistence, and Detection. Across three harnesses, six language models, and 14 model and harness configurations, attack success ranges from 12.6% to 80.9%, while Utility remains between 75.0% and 97.6%. Harness Configuration is the most vulnerable phase across all three harnesses, showing that attacks can succeed by altering security sensitive parameters within otherwise authorized workflows. We also find that explicit risk recognition does not reliably lead to safe action, as some configurations detect risks in more than 90% of runs while retaining substantial attack success. These results highlight the need to evaluate agent safety across multiple harness responsibilities and at the level of the deployed model and harness configuration.

Yajing Bai, Jinhao Duan, Jie Peng +4

University of North Carolina at Chapel Hill · University of Central Florida · Michigan State University

AI Agent Safety Benchmarks LLM Agent Safety LLM Agent Harnesses Prompt Injection Attacks on AI Agents Evaluation Awareness in Language Models

Ask PDF

Abstract

LLM Agent Harnesses

Explore similar work

Meta-Harness: End-to-End Optimization of Model Harnesses

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety