cs.AIOct 3, 2026

What Does a Harness Buy? Tokens, Mostly

Authors: Yangze Liu, Zhongyi Han

Organizations: Shandong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A coding agent is a language model wrapped in a harness: the system prompt, the tool set, and the context management that turn a chat model into something that can work inside a repository. Production harnesses ship releases daily, vendors advertise pass-rate gains from harness changes, and leaderboards mix harnesses freely. What is rarely measured is how much the harness itself moves the score when the model is held fixed. We run five models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified, and rerun the same configurations to calibrate how much a score moves when nothing changes but the run. On 447 tasks and the two models we ran there, Claude Code and mini-SWE-agent, the heaviest and the lightest harness, are equivalent within five points. On a 45-task hard subset and five models, swapping the harness flips as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness wins in one run are not the tasks it wins in the next. The one harness effect that clears the noise is a loss, not a gain: OpenCode trails by up to 9 points on the large pool, and on one model half of that gap sits in runs its output cap cut short. What the harness does decide is the bill. With the same model, the same tasks, and one price list, cost per task differs by up to 3x across harnesses. The gap is set at the first call, by the preamble of system prompt and tool schemas each harness sends with every step, and scaled by the number of steps; per-step growth and per-call tool output differ far less. The provider's price for cached input scales the bill and does not reorder it. The rerun data also give the resolution a harness comparison needs: at the discordance we observe, 45 tasks catch a 13-point gap only half the time and no gap with 80% power, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim.

Figures & tables

Figure 1: Same model, different harness. (a) Input tokens per task against pass rate on the 45 hard tasks, one run of every model under every harness. Within a model the harnesses spread two- to three-fold in tokens and stay inside the gap that 45 tasks catch only half the time. (b) Paired pass-rate difference for each pair of harnesses on each model and task set, with 95% intervals for the paired difference; vertical ticks mark the rerun range where a configuration was run more than once. Filled points are the pairs that clear the noise, and all of them are OpenCode trailing on the 447-task pool.
ModelPair A−BA-BnnAABBbbccΔ\Delta (pp)McNemar ppVerdict
Qwen3.6-35B-A3BCC −- mini4453103182634−- 1.8 [ −- 5.2, +1.6].37equivalent
 CC −- OC4453102974330+2.9 [ −- 0.9, +6.7].16inconclusive
 mini −- OC4453182974524+4.7 [+1.1, +8.4].02inferior BB
Qwen3.8-27BCC −- mini4443473531723−- 1.4 [ −- 4.2, +1.5].43equivalent
 CC −- OC4443473125217+7.9 [+4.3, +11.5]<< .001inferior BB
 mini −- OC4453533125716+9.2 [+5.5, +12.9]<< .001inferior BB
Table 1: Paired comparison on pool447. For each pair A−BA-B , bb is the number of tasks passed by AA only and cc by BB only. Δ\Delta is the paired difference with a Newcombe 95% interval, pp is McNemar’s exact test, unadjusted. The last column reads equivalent when the two one-sided tests at δ\delta = 5 points pass on the 90% interval, inferior when McNemar’s test rejects, and inconclusive otherwise.
ModelClaude Codemini-SWE-agentOpenCode (rg)
Qwen3.6-35B-A3B11/45  ×2{}^{\,\times 2} 24% [14, 39]15/45  ×2{}^{\,\times 2} 33% [21, 48]11/45  ×2{}^{\,\times 2} 24% [14, 39]
Qwen3.8-27B19/43  ×2{}^{\,\times 2} 44% [30, 59]17/45  ×2{}^{\,\times 2} 38% [25, 52]17/45  ×2{}^{\,\times 2} 38% [25, 52]
DeepSeek-V4-Flash21/45  ×2{}^{\,\times 2} 47% [33, 61]22/45  ×1{}^{\,\times 1} 49% [35, 63]19/45  ×1{}^{\,\times 1} 42% [29, 57]
GLM-5.3-Flash22/45  ×1{}^{\,\times 1} 49% [35, 63]20/45  ×1{}^{\,\times 1} 44% [31, 59]18/45  ×1{}^{\,\times 1} 40% [27, 55]
HY4-Preview23/45  ×1{}^{\,\times 1} 51% [37, 65]26/45  ×1{}^{\,\times 1} 58% [43, 71]21/45  ×1{}^{\,\times 1} 47% [33, 61]
Claude Opus 536/45  ×3{}^{\,\times 3} 80% [66, 89]——
Table 2: Pass rate on hard45. Each cell is the first run, as passed/45 with the Wilson 95% interval; the superscript is how many times that cell was run. Claude Opus 5 runs inside Claude Code only. Qwen3.8-27B in Claude Code has 43 scored tasks; two trials were lost to infrastructure. The pass count of every rerun is in Table A3 .
Figure 2: Swapping the harness flips as many tasks as rerunning the same harness. Each point is one pair of runs on the same tasks, and the value is the share of tasks whose outcome differs between the two. Rerun pairs share model, harness, and settings. Harness pairs share the model. Model pairs share the harness. Filled points are hard45, open points pool447; bars are medians.
Figure 3: How trials end. One bar per model and harness, split into pass, fail with the model stopping on its own, output-token cutoff, context overflow, step or time cap or crash, and trials lost to infrastructure, which are not scored. dsh is DeepSeek’s own harness.
Figure 4: What the model reads at each step. Input tokens of every call against the step index, one faint line per trial and a bold median over the trials still running at that step, drawn while at least half of them are. The harnesses differ first in where the line starts, which is the preamble, and less in how fast it climbs. The cliffs are compactions.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

 Claude Codemini-SWE-agentOpenCode
Version (pinned per arm) a2.1.259–2.1.273 (2.1.263 on GLM and Opus)2.4.61.18.30–1.18.31 (1.18.30 on GLM)
Samplingnone sent (no flag)none sentnone sent
Thinking / effortGLM: effort high (CLI flag); Qwen: effort xhigh via env (vLLM compat., a no-op) b ; DS: provider defaultGLM: reasoning_effort = high (request body); Qwen/DS: defaultGLM: reasoningEffort = high (model config); Qwen/DS: default
Toolsbuilt-in set (Bash, Read, Edit, Write, Grep…); WebSearch disallowedbash onlybash, read, edit, write, glob, grep, webfetch, task; ripgrep pre-placed (rg arms)
Context managementauto-compaction: 110k window (env override) on the Qwen arms, client default elsewhere cnone; per-observation head/tail trim (10k chars); hard stop at 131kprune (40k tool-output tail) + auto-compaction at limit.input minus a 20k reserve (78k); input limit 98k, output limit 32k
Termination--max-turns 300 ; soft timeout (exit 124)step_limit=300 ; no soft timeoutbuild.steps=300 ; soft timeout (exit 124)
Appendix
Table A1: What was held fixed. Verified from the harness configurations, the Harbor job files, and the proxy logs.
(a) Denominator ledger
SWE-bench Verified: 500 tasks
−- 8 psf/requests instances (verifier depends on httpbin.org , hangs offline) →\rightarrow pool492 == 492
→\rightarrow hard45 == 45 (whole difficulty bands 1–4 h and >> 4 h: 42 ++ 3; zero sampling degrees of freedom) ++ pool447 == 447 (the complementary remainder, disjoint)
django__django-11133 is filed under the hard45 label as the universal smoke task but belongs to pool447; it has no usable hard45 row, so the scored hard45 universe is 45.
(b) What the denominator loses, canonical run-1 trials
SubsetModelArmnnokvoidoverfl.t/outstepout-cutother
Appendix
Table A2: Denominator ledger. (a) From SWE-bench Verified to the two task sets. (b) For the canonical run of every cell, the trials lost to infrastructure (void, removed from the denominator) and the trials that ended at a limit (kept and scored as failures).
ModelSubsetArmRunnnokpassvoidStatusJobs
Qwen3.6hard45CC14545110completecc-qwen36-hard45-nw
  CC24545100completecc-qwen36-hard45-nw-k2
  mini14545150completemini-qwen36-hard45-nw (+ -r , -r2 )
  mini24545130completemini-qwen36-hard45-nw-k2
  OC-rg14545110completeoc-qwen36-hard45-nw-rg (+ -r )
  OC-rg24545110completeoc-qwen36-hard45-nw-rg-k2
Appendix
Table A3: Coverage matrix. Every cell of the grid, with the runs, the scored task count, the passes of each run, and the Harbor jobs it is assembled from.
Figure A1: hard45, task by task. Canonical run of every cell. Eight tasks are solved by no arm and four by every arm.
Figure A2: Smallest detectable paired difference against sample size. Exact McNemar test at α\alpha = .05 and 80% power, at the pooled discordance rate (solid line) and across the range observed between harness pairs (band); the asymptotic formula is dashed. At 45 tasks no difference reaches 80% power.
Figure A3: Where the bill comes from. Top: tokens per trial on DeepSeek-V4-Flash and GLM-5.3-Flash, split into cache-miss input, cache-hit input, and output including reasoning. Bottom: the same tokens priced at the vendor’s list price. Annotations give the cache-hit share, steps per trial, and input tokens per step for the three harnesses.
(a) Tool-return size per call (chars)
Harnesscallsmeanmedianp90Largest sources
Claude Code4,4031,2373923,095Bash 68%, Read 30%
mini-SWE-agent3,7871,1864613,121bash 100%
OpenCode2,6331,1033113,049read 49%, bash 39%, grep 9%
Appendix
Table A4: Where the per-step input goes. GLM-5.3-Flash on hard45, 45 runs per harness. (a) Size of a single tool return. (b) The per-step input budget, reconstructed from the trajectories and compared with the provider’s measured usage.
SettingCC versionSolved / 45flipsMcNemaroutputreas.stepsUSD/trial
  kk95% CI (%)/135pp vs. s1tok.tok. (mean)
Control, run 12.1.26336[66.2, 89.1]——15,9844,916341.83
Control, run 22.1.26335[63.7, 87.5]—1.0012,7775,927321.51
Control, run 32.1.26335[63.7, 87.5]—1.0014,9305,345391.60
Bash-only, run 12.1.26336[66.2, 89.1]16/1351.0013,5395,378311.42
Bash-only, run 22.1.26337[68.7, 90.7]7/1351.0012,2505,853281.63
Appendix
Table A5: Claude Code on Claude Opus 5, four settings, hard45. Flips are counted against the three control runs pooled. Cost is what Claude Code itself reports, at Anthropic’s list price.
Harnesssolved / attemptsper-task meanΔ\Deltaup /signmed. stepsmed. output
 onoffon →\to off(pp)downppon →\to offon →\to off
Claude Code21/9022/870.233 →\to 0.256+2.27 / 50.77103.5 →\to 12627,761 →\to 24,550
mini-SWE-agent28/9018/900.311 →\to 0.200−- 11.12 / 100.04106.5 →\to 9825,198 →\to 20,582
OpenCode15/8816/860.148 →\to 0.182+3.47 / 50.7756 →\to 78.514,018 →\to 15,949.5
HY4-Preview, Claude Code only, one run per side, thinking switched at the API
Claude Code23/4518/450.511 →\to 0.400−- 11.14 / 90.2775 →\to 9043,189 →\to 15,001
Appendix
Table A6: Thinking on and off, hard45. Top block: Qwen3.6-35B-A3B, three harnesses, two runs per side, thinking switched at the serving layer; the per-task mean is paired over tasks both sides scored, and pp is a two-sided sign test on the tasks whose mean changed. The OpenCode arms here run without ripgrep in place. Bottom block: HY4-Preview, Claude Code only, one run per side, thinking switched through the API’s thinking field; with one run per side the same sign test is McNemar’s exact test on the 9 lost and 4 gained tasks.
ModelPairbaseline Δ\Delta (pp) [95% CI] ( nn )no ovfl.per-prot.OC shiprun 2
hard45
Qwen3.6CC −- mini-8.9 [-23.9, +6.6] (45)-9.3 43-6.8 44—-6.7
 CC −- OC+0.0 [-11.8, +11.8] (45)==+0.0 44+11.1-2.2
 mini −- OC+8.9 [-4.0, +21.5] (45)+9.3 43==+20.0+4.4
Qwen3.8CC −- mini+7.0 [-5.5, +19.1] (43)+0.0 32+7.0—+4.4 45
 CC −- OC+7.0 [-1.8, +15.5] (43)+7.1 42+8.1 37+4.7+11.4 44
Appendix
Table A7: Sensitivity of every paired difference. Baseline as in Tables 2 and 1 ; the other columns recompute the difference after dropping overflowed trials, scoring per protocol, using OpenCode as shipped, and swapping in the second run. == means the variant removes no task and the value is unchanged.

Explore similar work

CardsList
Sep 8, 2026cs.AI

Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite

An agentic coding system couples a language model to a harness: the tools, prompts and control flow that turn a chat model into an autonomous software engineer. Vendors ship harnesses tuned to their own models, and practitioners assume the vendor-native pairing solves more tasks. We measure that assumption with paired same-model contrasts on a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks. The same 80 tasks ran under claude-agent-sdk and under deepagents on claude-opus-4-8, and under the openai-codex SDK and deepagents on gpt-5.5, with gemini-3.5-flash and deepseek-v3.2 as side cells. 792 of 800 planned runs were graded by an isolated oracle. Neither contrast resolves an average advantage for either harness: -1.25 pp for Opus 4.8 (48.8% vs 50.0%, task-bootstrap 95% CI [-10.0, +7.5]) and +1.25 pp for GPT-5.5 (55.6% vs 54.4%, CI [-4.4, +6.9]). The Opus average combines opposite strata: the native harness trails by 9.0 pp on the 61 repository tasks and leads by 23.7 pp on the 19 contest tasks (label-permutation p = 0.003). The partition was chosen after seeing the data and needs a designed replication. Correctness and completion also separate: 22 of 81 runs cancelled at the wall-clock ceiling had produced a passing patch. Re-priced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8 and 1.2 times on GPT-5.5. These are observed-usage estimates. On the Anthropic account 58 runs left no usage record, and allocating that spend to either cell would move the Opus ratio between 0.7 and 2.3, so the billed ordering is unresolved. This revision corrects an August 2026 manuscript whose cost figures rested on a usage-semantics defect in our own telemetry (Section 5.1). We release the orchestrator, grading oracle, reanalysis code and derived aggregates. The tasks stay private.

Mohsen Arjmandi

evolutionID GmbH, Germany

Coding AgentsLLM Agent EvaluationAI Agent BenchmarksLLM Agent Harnesses
AskPDF
Aug 26, 2026cs.AI

Same Model, Different Harness: Different Coding-Agent Results

A coding agent combines a model with a harness, which decides what the model sees, which tools it can use, and how the work continues. We ask whether changing the harness changes the result when the model and task stay fixed. We compare two configurations of the same harness on three coding benchmarks. The control supplies the full conversation in time order, while the treatment keeps the same record but mechanically shortens older tool results as the context fills and responds to repeated or stalled work. Under tight context, the treatment raises mean per-task fail-to-pass fraction (F2PF) in all three pressure comparisons and increases complete solutions on SWE-bench Verified and SWE-bench Pro. The tight-window Verified comparison uses 169 tasks, a 20,480-token window, and a fixed 480-second attempt endpoint; on this cohort, treatment raises mean per-task F2PF from 28 percent to 49 percent and complete solutions from 43 to 72. Without model-specific retuning, the same frozen treatment also raises both endpoints on the same cohort for three additional models with different designs. In the wide-window Qwen3.6 comparisons, observed arm outcomes are close on Verified and Pro, while FeatureBench retains a higher mean per-task F2PF under treatment. On the wide-window Verified cohort, treatment also serves fewer prompt tokens per turn. Because changing the harness changed what unchanged model weights could accomplish, coding-agent evaluations should treat the model and harness together as the tested solver.

Sydney Lewis

LLM Agent HarnessesAgent Harness OptimizationAI Agent Benchmarks
AskPDF
Oct 1, 2026cs.AI

Finding the Right Fit: Model-Harness Interactions across Agent Tasks

Choosing an agent system means choosing both a language model and the harness through which it acts. We ask whether a strong model, harness, or pairing stays strong when the setting changes. We evaluate 66 configurations: four configurable harnesses (OpenHands, DeepSeek Harness, PI, and openJiuwen) paired with five models on TUA-Bench, ALE-CLI, and Terminal-Bench 4, plus the native Codex-GPT and Claude Code-Claude pairings. Model rankings reverse across harnesses. On Terminal-Bench 4, Claude leads GPT by 7.94 points in OpenHands but trails it by 30.16 points in PI. For four of the five models, the best harness changes from one benchmark to another, yet some pairings hold: openJiuwen gives Kimi its highest score on all three benchmarks, by 5.61 to 11.11 points. A model's own vendor harness is not reliably its best, and higher cost does not reliably buy a higher score. On Terminal-Bench 4, GPT scores higher under PI than under DSH at less than a quarter of the cost per task. Matched trajectories suggest why fit varies. Models start almost all repairs themselves, so much depends on whether the harness hands failures back in a form the model can use. GPT does best with PI's lean scaffold, while Kimi, which often issues malformed tool calls, does best in openJiuwen. We argue that the model, the harness, and the task should be evaluated together, and we release the harness adapters, evaluation code, and all 6,204 scored trajectories at https://github.com/liyix/finding-the-right-fit and https://huggingface.co/datasets/yixuanli97/finding-the-right-fit.

Yixuan Li, Yiyun Zhou, Yao Long Teng +6

College of Computing and Data Science, Nanyang Technological University, Singapore

Tool-Augmented Language Model AgentsLLM Agent EvaluationAI Agent BenchmarksLLM Agent Harnesses
AskPDF