AI agent harnesses play the critical role of providing LLMs with an interface to a computing environment. Recent studies show that AI agents' task-execution accuracy depends mostly on the LLM used, while harness-side variances such as system prompts or tools have minimal impact. Rather, such harness-side variances show a larger impact on the cost of agent executions, achieving cost reductions up to 2.6 times on the same model and task. To further explore these harness-side cost optimization opportunities, we analyze the cost accumulation patterns across various agent traces and identify system opportunities to reduce token use and LLM calls without modifying the model or affecting task accuracy. Based on this analysis, we design Agency, a harness interposition platform that provides a unified abstraction and allows easy and portable implementation of harness-side system optimizations on existing off-the-shelf harnesses such as Claude Code or OpenAI Codex. Using Agency, we implement proof-of-concept optimization techniques based on our analysis and achieve cost reductions up to 35% across various real-world benchmarks on existing harnesses without compromising accuracy.
Figures & tables
Model
Input
Cache write
Cache read
Output
GPT-6-Luna*
0.11
0.1375
0.011
0.55
GPT-6.1-Sol*
2.20
2.75
0.11
11.00
Sonnet 5.5
2.20
2.75
0.11
11.00
Opus 5.5
4.40
5.50
0.22
22.00
Qwen3.8-27B
-
-
-
-
Table 1: Model prices in USD (1M). “Input” is uncached input.
Harness
System Prompt
Tool Schema (count)
Claude Code
4.1K
8.7K (18)
Codex
6.1K
5.3K (21)
Basic-full
3.4K
0.82K (6)
Basic-bash
0
0.24K (1)
Table 2: Baseline token usage and tool count across harnesses
Figure 1: Task accuracy (bars) and cost (dots) of each model and harness on the 100 tasks. Accuracy is the ratio of runs that passed.
Source (Num. Tasks)
Count
Avg. Length
Avg. Steps
SWE-bench (40)
1,480
19K-43K*
8-63*
SWE-bench Pro (40)
1,480
28K-62K*
10-72*
Terminal-Bench 2 (20)
740
23K-50K*
11-62*
OfficeQA Pro (100)
3,200
29K-47K*
10-17*
TraceLab
4,265
99K
84
FreeInference
14,036
140K
87
Table 3: Per-dataset trace count, average token length and steps.
Figure 2: LLM call cost breakdown on benchmark and public traces across different total token lengths.
Figure 3: Average total cost of traces by their total number of tool calls and tokens on benchmark and public traces (log scales).
Figure 4: Tool output token usage breakdown per tool type. Each bar averages over the benchmark traces.
Figure 5: Share of turns (bars) and cost (dots) spent on search, such as listing or pattern matching files.
Figure 6: Average breadth and subtask path size (left) and the average token and cost savings of parallel subtask pruning (right), by session context size. Breadth is the number of calls per depth of the graph, and path length is the number of depths.
Figure 8: Cost ratio (bars) and accuracy ratio (dots) of the combined techniques compared to baseline runs for each model and harness, averaged across tasks.
Figure 9: Cost ratio of the combined techniques compared to baseline runs, averaged across harnesses. OQ is OfficeQA Pro, in a flat directory or a tree, with plaintext or UUID file names.
Figure 10: Cost ratio distribution of the combined techniques compared to baseline per each task for each model and harness. The box spans the quartiles with the median as a line, and the whiskers the 5th and 95th percentiles.
Figure 11: Left: cost ratio of the combined techniques compared to baseline, with respect to baseline’s original number of tool calls and tokens per each task, averaged over all models and harnesses. Right: the context at each model call of a SWE-bench Pro task with Luna on Basic-full, for the baseline and the compacted run; triangles mark Subtask Stack Machine compactions.
Figure 12: Turn count when each technique fires in the combined techniques on Basic-full, using the share of the runs that reached the turn in which the technique acted. A darker cell is a larger share, scaled for each technique over both panels.
Cache read (K)
New input (K)
Output (K)
Steps
Cost (USD)
Cost
Technique
Tasks
Base
New
Base
New
Base
New
Base
New
Base
New
Ratio
Tool Output Compaction
Code
860
806
34
30
11.1
11.2
32.0
32.4
0.0202
0.0192
0.95
Context Garbage Collection
Code
860
667
34
34
11.1
10.3
32.0
32.5
0.0202
0.0177
0.88
Subtask Stack Machine
Code
860
644
34
42
11.1
11.2
32.0
32.6
0.0202
0.0190
0.94
Metadata-Assisted Search
OQ
567
330
48
35
5.4
4.3
16.7
13.5
0.0158
0.0108
0.68
Tool Call Batching
Code
860
816
34
40
11.1
11.1
32.0
26.0
0.0202
0.0206
1.02
Table 4: Cost optimization technique ablation using Basic-full harness with GPT-6-Luna, averaged over the task. Each technique is utilized in isolation. "Code" is the 100-task SWE-Bench, SWE-Bench Pro, and TerminalBench-2 set, and "OQ" is the 100-task OfficeQA set.