AI agent harnesses play the critical role of providing LLMs with an interface to a computing environment. Recent studies show that AI agents' task-execution accuracy depends mostly on the LLM used, while harness-side variances such as system prompts or tools have minimal impact. Rather, such harness-side variances show a larger impact on the cost of agent executions, achieving cost reductions up to 2.6 times on the same model and task. To further explore these harness-side cost optimization opportunities, we analyze the cost accumulation patterns across various agent traces and identify system opportunities to reduce token use and LLM calls without modifying the model or affecting task accuracy. Based on this analysis, we design Agency, a harness interposition platform that provides a unified abstraction and allows easy and portable implementation of harness-side system optimizations on existing off-the-shelf harnesses such as Claude Code or OpenAI Codex. Using Agency, we implement proof-of-concept optimization techniques based on our analysis and achieve cost reductions up to 35% across various real-world benchmarks on existing harnesses without compromising accuracy.
Figures & tables
Model
Input
Cache write
Cache read
Output
GPT-6-Luna*
0.11
0.1375
0.011
0.55
GPT-6.1-Sol*
2.20
2.75
0.11
11.00
Sonnet 5.5
2.20
2.75
0.11
11.00
Opus 5.5
4.40
5.50
0.22
22.00
Qwen3.8-27B
-
-
-
-
Table 1: Model prices in USD (1M). “Input” is uncached input.
Harness
System Prompt
Tool Schema (count)
Claude Code
4.1K
8.7K (18)
Codex
6.1K
5.3K (21)
Basic-full
3.4K
0.82K (6)
Basic-bash
0
0.24K (1)
Table 2: Baseline token usage and tool count across harnesses
Figure 1: Task accuracy (bars) and cost (dots) of each model and harness on the 100 tasks. Accuracy is the ratio of runs that passed.
Source (Num. Tasks)
Count
Avg. Length
Avg. Steps
SWE-bench (40)
1,480
19K-43K*
8-63*
SWE-bench Pro (40)
1,480
28K-62K*
10-72*
Terminal-Bench 2 (20)
740
23K-50K*
11-62*
OfficeQA Pro (100)
3,200
29K-47K*
10-17*
TraceLab
4,265
99K
84
FreeInference
14,036
140K
87
Table 3: Per-dataset trace count, average token length and steps.
Figure 2: LLM call cost breakdown on benchmark and public traces across different total token lengths.
Figure 3: Average total cost of traces by their total number of tool calls and tokens on benchmark and public traces (log scales).
Figure 4: Tool output token usage breakdown per tool type. Each bar averages over the benchmark traces.
Figure 5: Share of turns (bars) and cost (dots) spent on search, such as listing or pattern matching files.
Figure 6: Average breadth and subtask path size (left) and the average token and cost savings of parallel subtask pruning (right), by session context size. Breadth is the number of calls per depth of the graph, and path length is the number of depths.
Figure 8: Cost ratio (bars) and accuracy ratio (dots) of the combined techniques compared to baseline runs for each model and harness, averaged across tasks.
Figure 9: Cost ratio of the combined techniques compared to baseline runs, averaged across harnesses. OQ is OfficeQA Pro, in a flat directory or a tree, with plaintext or UUID file names.
Figure 10: Cost ratio distribution of the combined techniques compared to baseline per each task for each model and harness. The box spans the quartiles with the median as a line, and the whiskers the 5th and 95th percentiles.
Figure 11: Left: cost ratio of the combined techniques compared to baseline, with respect to baseline’s original number of tool calls and tokens per each task, averaged over all models and harnesses. Right: the context at each model call of a SWE-bench Pro task with Luna on Basic-full, for the baseline and the compacted run; triangles mark Subtask Stack Machine compactions.
Figure 12: Turn count when each technique fires in the combined techniques on Basic-full, using the share of the runs that reached the turn in which the technique acted. A darker cell is a larger share, scaled for each technique over both panels.
Cache read (K)
New input (K)
Output (K)
Steps
Cost (USD)
Cost
Technique
Tasks
Base
New
Base
New
Base
New
Base
New
Base
New
Ratio
Tool Output Compaction
Code
860
806
34
30
11.1
11.2
32.0
32.4
0.0202
0.0192
0.95
Context Garbage Collection
Code
860
667
34
34
11.1
10.3
32.0
32.5
0.0202
0.0177
0.88
Subtask Stack Machine
Code
860
644
34
42
11.1
11.2
32.0
32.6
0.0202
0.0190
0.94
Metadata-Assisted Search
OQ
567
330
48
35
5.4
4.3
16.7
13.5
0.0158
0.0108
0.68
Tool Call Batching
Code
860
816
34
40
11.1
11.1
32.0
26.0
0.0202
0.0206
1.02
Table 4: Cost optimization technique ablation using Basic-full harness with GPT-6-Luna, averaged over the task. Each technique is utilized in isolation. "Code" is the 100-task SWE-Bench, SWE-Bench Pro, and TerminalBench-2 set, and "OQ" is the 100-task OfficeQA set.
As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.
An LLM agent's cost and success depend on its model, its harness, and its environment. In addition to the widely studied directions of parameter tuning and harness optimization, we propose the third direction: optimization over stationary environments. Agents are often deployed in rich environments shared by many tasks, such as a company's Slack workspace or a complex collection of data sources, documents or tickets. We call these environments stationary environments. We introduce EnvOpt, a method for automated end-to-end optimization of stationary environments for any given model and harness, using real or synthetic tasks without needing any graders or reference answers. On AutomationBench, EnvOpt reduces the cost of Claude Haiku 4.5 by 72.7% and raises its pass rate by 14.34%. The same environment transfers to Opus 5, cutting its cost by 68.1% with a 1.34% gain. Added on top of Meta-Harness's optimized harness for Haiku 4.5, it lowers cost by a further 83.4% and adds 11.17% accuracy over Meta-Harness alone. Across 5 benchmarks, EnvOpt significantly reduces the cost of agents while improving or maintaining pass rate, and is transferable to different models and different harnesses. We also discuss common successful and rejected environment optimization strategies during automated optimization. More broadly, our results show that how information is represented to an agent shapes both its cost and its success, making the environment worth optimizing in addition to the parameters and the harness.
Mingxuan Li, Huanzhi Mao, Qiuyang Mang +3
University of California, Berkeley · University of Chicago · Bespoke Labs
Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro
Zhuoshi Pan, ZhiYuan Feng, Yukuo Cen +6
Tsinghua University · Z.AI Co., Ltd. · Shanghai Jiao Tong University +2