Same Model, Different Harness: Different Coding-Agent Results

Sydney Lewis

cs.AIAug 26, 2026

Same Model, Different Harness: Different Coding-Agent Results

Authors: Sydney Lewis

Abstract

A coding agent combines a model with a harness, which decides what the model sees, which tools it can use, and how the work continues. We ask whether changing the harness changes the result when the model and task stay fixed. We compare two configurations of the same harness on three coding benchmarks. The control supplies the full conversation in time order, while the treatment keeps the same record but mechanically shortens older tool results as the context fills and responds to repeated or stalled work. Under tight context, the treatment raises mean per-task fail-to-pass fraction (F2PF) in all three pressure comparisons and increases complete solutions on SWE-bench Verified and SWE-bench Pro. The tight-window Verified comparison uses 169 tasks, a 20,480-token window, and a fixed 480-second attempt endpoint; on this cohort, treatment raises mean per-task F2PF from 28 percent to 49 percent and complete solutions from 43 to 72. Without model-specific retuning, the same frozen treatment also raises both endpoints on the same cohort for three additional models with different designs. In the wide-window Qwen3.6 comparisons, observed arm outcomes are close on Verified and Pro, while FeatureBench retains a higher mean per-task F2PF under treatment. On the wide-window Verified cohort, treatment also serves fewer prompt tokens per turn. Because changing the harness changed what unchanged model weights could accomplish, coding-agent evaluations should treat the model and harness together as the tested solver.

Figures & tables

Figure 1: The closed loop. The harness retains the in-memory conversation but rebuilds the model’s current view for each step. Recorded execution patterns can change a later step. Separate files preserve the complete run record.

Figure 2: The treatment’s working-view rule. The treatment starts shortening only after the estimated full prompt reaches half the configured window. The newest four tool results remain full. Older results keep their beginning and end within the printed caps, while the in-memory result remains complete. Capped bar lengths use a base-two logarithmic scale. The printed values are exact.

Figure 3: The primary paired experiment. Both arms use the same model-facing harness code, model, tasks, context capacity, run protocol, serving setup, and evaluator. Configuration layers enable the treatment condition on the treatment arm.

Figure 4: A real Verified treatment trace. The lower-left panel shows an excerpt from the exact 512-character view of a 26,684-character tool result. The full result remained saved. The lower-right panel shows assembled prompt tokens before and after shortening began at turn 60. The top row reports the recorded detector fact and the reminder that followed.

Cell (context window)	Active runs	First turn	Typical reduction
Verified 43,008	107/169 (63.3%)	23 [11, 41]	52.9% [41.3, 64.8]
Verified 20,480	140/169 (82.8%)	10 [8, 19]	45.2% [33.1, 52.0]
Pro 49,152	246/316 (77.8%)	13 [8, 22]	64.4% [55.6, 71.3]
FeatureBench 47,104	143/183 (78.1%)	9 [7, 18]	55.7% [42.6, 64.6]

Table 1: Shortening activity in each primary treatment population. The Active runs column counts sessions with at least one shortening event. The First turn column marks when shortening first removes characters. The Typical reduction column reports the median share of characters removed. Brackets give the middle half of active runs.

Figure 5: The ten-test example. The patch fixes seven target tests, so its F2PF is 0.70. Three target tests still fail, so the task is not resolved. F2PF records useful movement below the all-or-nothing finish line.

benchmark	mean per-task F2PF C $\rightarrow$ T	complete solutions C $\rightarrow$ T
SWE-bench Verified	28% $\rightarrow$ 49%	43 $\rightarrow$ 72
SWE-bench Pro	15% $\rightarrow$ 33%	31 $\rightarrow$ 72
FeatureBench	11% $\rightarrow$ 20%	2 $\rightarrow$ 3

Table 2: Outcomes for the three primary Qwen3.6 pressure comparisons. Each row gives mean per-task F2PF and complete solutions for both arms. We round percentages to whole numbers. C and T denote control and treatment.

Figure 6: Qwen3.6 outcomes across the three Verified context windows at the completed-run endpoint under the fixed 480-second budget. Every point uses the same 169 tasks. The mean per-task F2PF label above each window gives the paired treatment-minus-control difference in percentage points.

Figure 7: One operational F2PF state per task under context pressure and with context effectively unconstrained at 262,144 tokens. Hatched segments show selected records with no F2P denominator. The fixed scoring rule assigns those records zero. Gray segments show measured zeros with a positive denominator. We round segment percentages independently. Each panel uses the task outcomes for the matching benchmark and window in Appendix Table 8 .

Figure 8: The fitted reading boundary for the three Qwen3.6 pressure comparisons. Horizontal lines are 95% repository-cluster bootstrap intervals, and the dotted line marks demand equal to one pressured window. A point farther right means that the model can complete more reading before half the runs stop.

model	design	F2PF C $\rightarrow$ T (gain; $\times$ )	solutions C $\rightarrow$ T (gain; $\times$ )
Qwen3.6 [ 3 ]	DeltaNet/attention MoE	28% $\rightarrow$ 49% (+21; 1.8 $\times$ )	43 $\rightarrow$ 72 (+29; 1.7 $\times$ )
Devstral [ 9 ]	dense transformer	17% $\rightarrow$ 37% (+20; 2.1 $\times$ )	22 $\rightarrow$ 53 (+31; 2.4 $\times$ )
Nemotron [ 10 ]	Mamba-2/attention MoE	12% $\rightarrow$ 18% (+6; 1.5 $\times$ )	16 $\rightarrow$ 25 (+9; 1.6 $\times$ )
Qwen3.8 [ 11 ]	dense DeltaNet/attention	20% $\rightarrow$ 35% (+15; 1.7 $\times$ )	32 $\rightarrow$ 54 (+22; 1.7 $\times$ )

Table 3: Paired outcomes on the same 169-task SWE-bench Verified cohort at 20,480 tokens and a fixed 480-second attempt budget. C and T denote control and treatment. The design column uses the term mixture of experts (MoE). Parentheses give the absolute gain and T/C multiplier; F2PF gains are percentage points.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

benchmark	$n$	selection rule
SWE-bench Verified	169	The original pressure pool contained 187 tasks. We applied the fixed 480-second cap to both arms and retained the 169 tasks that produced complete paired outcomes within that cap. We use the same task IDs at every Verified window and on every model.
SWE-bench Pro	316	We select tasks whose earlier large-window control run crossed the frozen reading-demand threshold.
FeatureBench	183	We use every canonical task runnable in the evaluation setup. Reading demand does not determine membership.

Appendix

Table 4: Cohorts used in the paired comparisons.

item	fixed setting
Cohort	We use the same 169 task IDs at every window.
Endpoint	We select the best finalized patch within the fixed 480-second budget.
Context windows	We use 20,480, 43,008, and 262,144 tokens.
Within a campaign	Control and treatment share one model-facing code state.
Across windows	We collected campaigns successively. The ordering describes the observed context regimes.

Appendix

Table 5: Fixed design for the matched-window Verified comparison.

window	F2PF difference [95% interval]	resolution difference [95% interval]
20,480 (tight)	+21.1 [+14.1, +28.3]	+17.2 [+10.7, +23.7]
43,008 (pressured)	+6.4 [+0.5, +12.4]	+6.5 [+0.6, +13.0]
262,144 (effectively unconstrained)	-0.3 [-4.5, +3.9]	-0.6 [-5.9, +4.7]

Appendix

Table 6: Completed-run paired differences on the same 169-task Verified cohort under the fixed 480-second budget. Each entry gives treatment minus control in percentage points, with paired-task bootstrap 95% intervals.

benchmark	window	$n$	solutions C	solutions T	T-only:C-only ( $p$ )
SWE-bench Verified	20,480	169	43	72	34:5 ( $<0.0001$ )
SWE-bench Pro	49,152	316	31	72	47:6 ( $<0.0001$ )
FeatureBench	47,104	183	2	3	1:0 ( $1$ )
SWE-bench Verified	43,008	169	76	87	20:9 ( $0.0614$ )
SWE-bench Verified (common cohort)	262,144	169	102	101	9:10 ( $1$ )
SWE-bench Pro	262,144	316	101	99	11:13 ( $0.8388$ )

Appendix

Table 7: Complete solutions in the paired Qwen3.6 comparisons. C and T denote control and treatment. The pressure rows are primary. The remaining rows report observed results at larger context windows. Tests compare arms within each row. The Verified wide-window row is a post-hoc common-cohort projection.

benchmark	window	F2PF C (%)	F2PF T (%)	ratio	higher T:C:tie ( $p$ )
SWE-bench Verified	20,480	28.0	49.1	1.75	42:6:121 ( $<0.0001$ )
SWE-bench Pro	49,152	15.2	32.7	2.15	84:13:219 ( $<0.0001$ )
FeatureBench	47,104	10.5	19.6	1.86	47:9:127 ( $<0.0001$ )
SWE-bench Verified	43,008	53.8	60.1	1.12	21:10:138 ( $0.0708$ )
SWE-bench Verified (common cohort)	262,144	69.0	68.7	1.00	9:8:152 ( $1$ )
SWE-bench Pro	262,144	44.8	44.5	0.99	22:24:270 ( $0.883$ )

Appendix

Table 8: Mean per-task F2PF in the same paired Qwen3.6 comparisons. C and T denote control and treatment. The final column gives task-level direction and its exact paired sign-test

p

-value.

benchmark	$n$	model turns C / T	prompt tokens (millions) C / T	solver wall time (hours) C / T
SWE-bench Verified	169	3,280 / 6,517	37.8 / 80.7	1.3 / 4.6
SWE-bench Pro	316	8,357 / 25,750	210.6 / 657.1	5.4 / 21.8
FeatureBench	183	5,281 / 15,178	140.3 / 412.8	2.5 / 14.7

Appendix

Table 9: Recorded model work for the three primary Qwen3.6 pressure comparisons. Each total covers the complete fixed run protocol used for the final outcome. Model turns count unique recorded turns within each solver invocation, while prompt tokens count the assembled model inputs at those turns. C/T denotes control/treatment.

measure	paired tasks	T/C ratio [95% interval]
Prompt tokens per task	154	0.959 [0.856, 1.071]
Prompt tokens per turn	153	0.928 [0.878, 0.979]

Appendix

Table 10: Context use in the 262,144-token Verified comparison. Each point estimate is the geometric mean of paired treatment/control task ratios. We build the intervals from 10,000 paired task-bootstrap samples. A ratio below one means that treatment served less context.

model	F2PF C/T (%)	difference [95% interval]	$p$	$p_{\mathrm{Holm}}$	solutions C/T	$p_{\mathrm{McNemar}}$
Devstral	17.2/36.8	+19.5 [+12.0, +26.9]	$<0.0001$	$<0.0001$	22/53	$<0.0001$
Nemotron	12.0/18.3	+6.4 [+1.4, +11.4]	$0.0169$	$0.0169$	16/25	$0.0636$
Qwen3.8	20.4/35.3	+14.9 [+9.9, +20.4]	$<0.0001$	–	32/54	$<0.0001$

Appendix

Table 11: Transfer-model outcome comparisons on the frozen 169-task SWE-bench Verified pressure cohort at a 20,480-token window and the fixed 480-second attempt endpoint. F2PF differences are treatment minus control in percentage points. Paired task bootstraps give 95% intervals in the same units. Exact paired sign tests compare F2PF directions, while exact paired McNemar tests compare resolution as a secondary outcome. The Holm column applies to the pre-registered Devstral–Nemotron family, whereas the separately registered Qwen3.8 comparison is outside that family.

benchmark	repositories	F2PF difference [95% interval]	higher T:C:tie ( $p_{\mathrm{Holm}}$ )
SWE-bench Verified	11	21.1 [13.7, 25.0]	7:0:4 ( $0.0469$ )
SWE-bench Pro	11	17.5 [11.8, 24.3]	11:0:0 ( $0.00586$ )
FeatureBench	22	9.1 [5.8, 14.1]	16:2:4 ( $0.00586$ )

Appendix

Table 12: Repository-cluster sensitivity of mean per-task F2PF in the three primary Qwen3.6 pressure comparisons. Differences are treatment minus control in percentage points. The final column gives repository-level direction and the Holm-adjusted exact sign-test

p

-value. The adjustment covers the six F2PF and resolution tests.

benchmark	repositories	resolution difference [95% interval]	higher T:C:tie ( $p_{\mathrm{Holm}}$ )
SWE-bench Verified	11	17.2 [9.6, 20.1]	6:0:5 ( $0.0625$ )
SWE-bench Pro	11	13.0 [8.2, 19.1]	11:0:0 ( $0.00586$ )
FeatureBench	22	0.5 [0.0, 1.5]	1:0:21 ( $1$ )

Appendix

Table 13: Repository-cluster sensitivity of complete resolution in the same comparisons. Differences are treatment minus control in percentage points. The final column uses the same six-test Holm family as Table 12 .

item	specification
Fit	We fit a logistic regression of stopping while reading on $\log_{2}(\text{demand}/\text{window})$ .
Interval	We use the 2.5th to 97.5th percentile of the bootstrap distribution.
Resampling	We draw 2,000 repository samples with replacement, using seed 20260806.
Excluded draws	We exclude draws with one outcome class, a singular fit, or a non-positive fitted slope.
Paired ratio	We use the same repository sample for both arms.
Valid ratios	All 2,000 paired samples produced estimates on all three benchmarks.

Appendix

Table 14: Reading-boundary fit and uncertainty calculation.

benchmark	$n$	control location	treatment location	stretch
SWE-bench Verified	166	0.71 [0.59, 0.84]	1.81 [1.43, 2.21]	2.53 [1.81, 3.59]
SWE-bench Pro	315	0.97 [0.90, 1.04]	2.15 [1.72, 3.23]	2.21 [1.80, 3.25]
FeatureBench	170	0.93 [0.82, 1.01]	2.09 [1.74, 2.34]	2.26 [1.86, 2.56]

Appendix

Table 15: Fitted Qwen3.6 reading boundaries. Each location gives the demand-to-window ratio where half the runs stop before the first source change. Brackets give repository-cluster bootstrap 95% intervals. Stretch equals the treatment location divided by the control location.

Cell	$n$	command	errors	condense	decay
Verified (20k)	169	484/117	313/105	0/0	4079/140
Verified (43k)	169	786/150	435/142	0/0	3212/107
Pro (49k)	316	6926/236	6181/230	519/104	9318/246
FeatureBench (47k)	183	1250/131	1017/127	44/16	5057/143

Appendix

Table 16: Observed context and command activity in the paired populations used for this analysis. Each entry after

n

gives the event count followed by the number of sessions with at least one event; all listed markers are zero in the corresponding control sessions.

Cell	checks	valid	setup	positive	applies	restores
Verified (20k)	2193/168	2193/168	0/0	45/31	56/31	24/21
Verified (43k)	2895/169	2895/169	0/0	50/36	62/36	32/29
Pro (49k)	10109/314	10109/314	0/0	2464/153	633/153	240/145
FeatureBench (47k)	3490/181	3490/181	0/0	424/84	262/84	90/75

Appendix

Table 17: Observed detector and conditioned-action activity in the same paired populations. Each entry gives the event count followed by the number of sessions with at least one event. The applies count excludes later restorations; all listed markers are zero in the corresponding control sessions.

Figure 9: Assembled prompt length by turn for six selected task pairs, two per benchmark. The horizontal reference is the configured window. Panel subtitles report control followed by treatment. The Verified examples are treatment-only resolutions. The Pro and FeatureBench examples have large positive paired F2PF differences.

family	model	policy setting	solutions / $n$	mean per-task F2PF (%)
R6 depth	Qwen3.6	frozen caps, prior run	6/15	45.3
		frozen caps, rerun	7/15	47.5
		all caps $\times$ 0.5	5/15	45.0
		all caps $\times$ 2.0	4/15	33.3
R7/R8 timing	Qwen3.6	activation 0.50, protect 4	3/15	22.2
		activation 0.25, protect 4	2/15	13.3

Appendix

Table 18: Exploratory single-run probes of context depth and activation timing. Rows compare only with the frozen setting for the same model and cohort.

Explore similar work

Oct 3, 2026cs.AI

What Does a Harness Buy? Tokens, Mostly

A coding agent is a language model wrapped in a harness: the system prompt, the tool set, and the context management that turn a chat model into something that can work inside a repository. Production harnesses ship releases daily, vendors advertise pass-rate gains from harness changes, and leaderboards mix harnesses freely. What is rarely measured is how much the harness itself moves the score when the model is held fixed. We run five models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified, and rerun the same configurations to calibrate how much a score moves when nothing changes but the run. On 447 tasks and the two models we ran there, Claude Code and mini-SWE-agent, the heaviest and the lightest harness, are equivalent within five points. On a 45-task hard subset and five models, swapping the harness flips as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness wins in one run are not the tasks it wins in the next. The one harness effect that clears the noise is a loss, not a gain: OpenCode trails by up to 9 points on the large pool, and on one model half of that gap sits in runs its output cap cut short. What the harness does decide is the bill. With the same model, the same tasks, and one price list, cost per task differs by up to 3x across harnesses. The gap is set at the first call, by the preamble of system prompt and tool schemas each harness sends with every step, and scaled by the number of steps; per-step growth and per-call tool output differ far less. The provider's price for cached input scales the bill and does not reorder it. The rerun data also give the resolution a harness comparison needs: at the discordance we observe, 45 tasks catch a 13-point gap only half the time and no gap with 80% power, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim.

Yangze Liu, Zhongyi Han

Shandong University

LLM Agent Harnesses AI Coding Agents LLM Agent Evaluation LLM Agents

Ask PDF

Sep 8, 2026cs.AI

Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite

An agentic coding system couples a language model to a harness: the tools, prompts and control flow that turn a chat model into an autonomous software engineer. Vendors ship harnesses tuned to their own models, and practitioners assume the vendor-native pairing solves more tasks. We measure that assumption with paired same-model contrasts on a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks. The same 80 tasks ran under claude-agent-sdk and under deepagents on claude-opus-4-8, and under the openai-codex SDK and deepagents on gpt-5.5, with gemini-3.5-flash and deepseek-v3.2 as side cells. 792 of 800 planned runs were graded by an isolated oracle. Neither contrast resolves an average advantage for either harness: -1.25 pp for Opus 4.8 (48.8% vs 50.0%, task-bootstrap 95% CI [-10.0, +7.5]) and +1.25 pp for GPT-5.5 (55.6% vs 54.4%, CI [-4.4, +6.9]). The Opus average combines opposite strata: the native harness trails by 9.0 pp on the 61 repository tasks and leads by 23.7 pp on the 19 contest tasks (label-permutation p = 0.003). The partition was chosen after seeing the data and needs a designed replication. Correctness and completion also separate: 22 of 81 runs cancelled at the wall-clock ceiling had produced a passing patch. Re-priced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8 and 1.2 times on GPT-5.5. These are observed-usage estimates. On the Anthropic account 58 runs left no usage record, and allocating that spend to either cell would move the Opus ratio between 0.7 and 2.3, so the billed ordering is unresolved. This revision corrects an August 2026 manuscript whose cost figures rested on a usage-semantics defect in our own telemetry (Section 5.1). We release the orchestrator, grading oracle, reanalysis code and derived aggregates. The tasks stay private.

Mohsen Arjmandi

evolutionID GmbH, Germany

Coding Agents LLM Agent Evaluation AI Agent Benchmarks LLM Agent Harnesses

Ask PDF

Sep 17, 2026cs.AI

An Empirical Study of Harness Design for Coding Agents

Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.

Run-Ze Fan, Zihao Zhang, Simin Ma +6

UMass Amherst · Emory University · UNC Charlotte

Coding Agents Agent Harness Optimization Tool-Using Agents LLM Context Management LLM Agent Planning

Ask PDF