Same Model, Different Harness: Different Coding-Agent Results

Sydney Lewis

cs.AIAug 26, 2026

Same Model, Different Harness: Different Coding-Agent Results

Authors: Sydney Lewis

Abstract

A coding agent combines a model with a harness, which decides what the model sees, which tools it can use, and how the work continues. We ask whether changing the harness changes the result when the model and task stay fixed. We compare two configurations of the same harness on three coding benchmarks. The control supplies the full conversation in time order, while the treatment keeps the same record but mechanically shortens older tool results as the context fills and responds to repeated or stalled work. Under tight context, the treatment raises mean per-task fail-to-pass fraction (F2PF) in all three pressure comparisons and increases complete solutions on SWE-bench Verified and SWE-bench Pro. The tight-window Verified comparison uses 169 tasks, a 20,480-token window, and a fixed 480-second attempt endpoint; on this cohort, treatment raises mean per-task F2PF from 28 percent to 49 percent and complete solutions from 43 to 72. Without model-specific retuning, the same frozen treatment also raises both endpoints on the same cohort for three additional models with different designs. In the wide-window Qwen3.6 comparisons, observed arm outcomes are close on Verified and Pro, while FeatureBench retains a higher mean per-task F2PF under treatment. On the wide-window Verified cohort, treatment also serves fewer prompt tokens per turn. Because changing the harness changed what unchanged model weights could accomplish, coding-agent evaluations should treat the model and harness together as the tested solver.

Figures & tables

Figure 1: The closed loop. The harness retains the in-memory conversation but rebuilds the model’s current view for each step. Recorded execution patterns can change a later step. Separate files preserve the complete run record.

Figure 2: The treatment’s working-view rule. The treatment starts shortening only after the estimated full prompt reaches half the configured window. The newest four tool results remain full. Older results keep their beginning and end within the printed caps, while the in-memory result remains complete. Capped bar lengths use a base-two logarithmic scale. The printed values are exact.

Figure 3: The primary paired experiment. Both arms use the same model-facing harness code, model, tasks, context capacity, run protocol, serving setup, and evaluator. Configuration layers enable the treatment condition on the treatment arm.

Figure 4: A real Verified treatment trace. The lower-left panel shows an excerpt from the exact 512-character view of a 26,684-character tool result. The full result remained saved. The lower-right panel shows assembled prompt tokens before and after shortening began at turn 60. The top row reports the recorded detector fact and the reminder that followed.

Cell (context window)	Active runs	First turn	Typical reduction
Verified 43,008	107/169 (63.3%)	23 [11, 41]	52.9% [41.3, 64.8]
Verified 20,480	140/169 (82.8%)	10 [8, 19]	45.2% [33.1, 52.0]
Pro 49,152	246/316 (77.8%)	13 [8, 22]	64.4% [55.6, 71.3]
FeatureBench 47,104	143/183 (78.1%)	9 [7, 18]	55.7% [42.6, 64.6]

Table 1: Shortening activity in each primary treatment population. The Active runs column counts sessions with at least one shortening event. The First turn column marks when shortening first removes characters. The Typical reduction column reports the median share of characters removed. Brackets give the middle half of active runs.

Figure 5: The ten-test example. The patch fixes seven target tests, so its F2PF is 0.70. Three target tests still fail, so the task is not resolved. F2PF records useful movement below the all-or-nothing finish line.

benchmark	mean per-task F2PF C $\rightarrow$ T	complete solutions C $\rightarrow$ T
SWE-bench Verified	28% $\rightarrow$ 49%	43 $\rightarrow$ 72
SWE-bench Pro	15% $\rightarrow$ 33%	31 $\rightarrow$ 72
FeatureBench	11% $\rightarrow$ 20%	2 $\rightarrow$ 3

Table 2: Outcomes for the three primary Qwen3.6 pressure comparisons. Each row gives mean per-task F2PF and complete solutions for both arms. We round percentages to whole numbers. C and T denote control and treatment.

Figure 6: Qwen3.6 outcomes across the three Verified context windows at the completed-run endpoint under the fixed 480-second budget. Every point uses the same 169 tasks. The mean per-task F2PF label above each window gives the paired treatment-minus-control difference in percentage points.

Figure 7: One operational F2PF state per task under context pressure and with context effectively unconstrained at 262,144 tokens. Hatched segments show selected records with no F2P denominator. The fixed scoring rule assigns those records zero. Gray segments show measured zeros with a positive denominator. We round segment percentages independently. Each panel uses the task outcomes for the matching benchmark and window in Appendix Table 8 .

Figure 8: The fitted reading boundary for the three Qwen3.6 pressure comparisons. Horizontal lines are 95% repository-cluster bootstrap intervals, and the dotted line marks demand equal to one pressured window. A point farther right means that the model can complete more reading before half the runs stop.

model	design	F2PF C $\rightarrow$ T (gain; $\times$ )	solutions C $\rightarrow$ T (gain; $\times$ )
Qwen3.6 [ 3 ]	DeltaNet/attention MoE	28% $\rightarrow$ 49% (+21; 1.8 $\times$ )	43 $\rightarrow$ 72 (+29; 1.7 $\times$ )
Devstral [ 9 ]	dense transformer	17% $\rightarrow$ 37% (+20; 2.1 $\times$ )	22 $\rightarrow$ 53 (+31; 2.4 $\times$ )
Nemotron [ 10 ]	Mamba-2/attention MoE	12% $\rightarrow$ 18% (+6; 1.5 $\times$ )	16 $\rightarrow$ 25 (+9; 1.6 $\times$ )
Qwen3.8 [ 11 ]	dense DeltaNet/attention	20% $\rightarrow$ 35% (+15; 1.7 $\times$ )	32 $\rightarrow$ 54 (+22; 1.7 $\times$ )

Table 3: Paired outcomes on the same 169-task SWE-bench Verified cohort at 20,480 tokens and a fixed 480-second attempt budget. C and T denote control and treatment. The design column uses the term mixture of experts (MoE). Parentheses give the absolute gain and T/C multiplier; F2PF gains are percentage points.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

benchmark	$n$	selection rule
SWE-bench Verified	169	The original pressure pool contained 187 tasks. We applied the fixed 480-second cap to both arms and retained the 169 tasks that produced complete paired outcomes within that cap. We use the same task IDs at every Verified window and on every model.
SWE-bench Pro	316	We select tasks whose earlier large-window control run crossed the frozen reading-demand threshold.
FeatureBench	183	We use every canonical task runnable in the evaluation setup. Reading demand does not determine membership.

Appendix

Table 4: Cohorts used in the paired comparisons.

item	fixed setting
Cohort	We use the same 169 task IDs at every window.
Endpoint	We select the best finalized patch within the fixed 480-second budget.
Context windows	We use 20,480, 43,008, and 262,144 tokens.
Within a campaign	Control and treatment share one model-facing code state.
Across windows	We collected campaigns successively. The ordering describes the observed context regimes.

Appendix

Table 5: Fixed design for the matched-window Verified comparison.

window	F2PF difference [95% interval]	resolution difference [95% interval]
20,480 (tight)	+21.1 [+14.1, +28.3]	+17.2 [+10.7, +23.7]
43,008 (pressured)	+6.4 [+0.5, +12.4]	+6.5 [+0.6, +13.0]
262,144 (effectively unconstrained)	-0.3 [-4.5, +3.9]	-0.6 [-5.9, +4.7]

Appendix

Table 6: Completed-run paired differences on the same 169-task Verified cohort under the fixed 480-second budget. Each entry gives treatment minus control in percentage points, with paired-task bootstrap 95% intervals.

benchmark	window	$n$	solutions C	solutions T	T-only:C-only ( $p$ )
SWE-bench Verified	20,480	169	43	72	34:5 ( $<0.0001$ )
SWE-bench Pro	49,152	316	31	72	47:6 ( $<0.0001$ )
FeatureBench	47,104	183	2	3	1:0 ( $1$ )
SWE-bench Verified	43,008	169	76	87	20:9 ( $0.0614$ )
SWE-bench Verified (common cohort)	262,144	169	102	101	9:10 ( $1$ )
SWE-bench Pro	262,144	316	101	99	11:13 ( $0.8388$ )

Appendix

Table 7: Complete solutions in the paired Qwen3.6 comparisons. C and T denote control and treatment. The pressure rows are primary. The remaining rows report observed results at larger context windows. Tests compare arms within each row. The Verified wide-window row is a post-hoc common-cohort projection.

benchmark	window	F2PF C (%)	F2PF T (%)	ratio	higher T:C:tie ( $p$ )
SWE-bench Verified	20,480	28.0	49.1	1.75	42:6:121 ( $<0.0001$ )
SWE-bench Pro	49,152	15.2	32.7	2.15	84:13:219 ( $<0.0001$ )
FeatureBench	47,104	10.5	19.6	1.86	47:9:127 ( $<0.0001$ )
SWE-bench Verified	43,008	53.8	60.1	1.12	21:10:138 ( $0.0708$ )
SWE-bench Verified (common cohort)	262,144	69.0	68.7	1.00	9:8:152 ( $1$ )
SWE-bench Pro	262,144	44.8	44.5	0.99	22:24:270 ( $0.883$ )

Appendix

Table 8: Mean per-task F2PF in the same paired Qwen3.6 comparisons. C and T denote control and treatment. The final column gives task-level direction and its exact paired sign-test

p

-value.

benchmark	$n$	model turns C / T	prompt tokens (millions) C / T	solver wall time (hours) C / T
SWE-bench Verified	169	3,280 / 6,517	37.8 / 80.7	1.3 / 4.6
SWE-bench Pro	316	8,357 / 25,750	210.6 / 657.1	5.4 / 21.8
FeatureBench	183	5,281 / 15,178	140.3 / 412.8	2.5 / 14.7

Appendix

Table 9: Recorded model work for the three primary Qwen3.6 pressure comparisons. Each total covers the complete fixed run protocol used for the final outcome. Model turns count unique recorded turns within each solver invocation, while prompt tokens count the assembled model inputs at those turns. C/T denotes control/treatment.

measure	paired tasks	T/C ratio [95% interval]
Prompt tokens per task	154	0.959 [0.856, 1.071]
Prompt tokens per turn	153	0.928 [0.878, 0.979]

Appendix

Table 10: Context use in the 262,144-token Verified comparison. Each point estimate is the geometric mean of paired treatment/control task ratios. We build the intervals from 10,000 paired task-bootstrap samples. A ratio below one means that treatment served less context.

model	F2PF C/T (%)	difference [95% interval]	$p$	$p_{\mathrm{Holm}}$	solutions C/T	$p_{\mathrm{McNemar}}$
Devstral	17.2/36.8	+19.5 [+12.0, +26.9]	$<0.0001$	$<0.0001$	22/53	$<0.0001$
Nemotron	12.0/18.3	+6.4 [+1.4, +11.4]	$0.0169$	$0.0169$	16/25	$0.0636$
Qwen3.8	20.4/35.3	+14.9 [+9.9, +20.4]	$<0.0001$	–	32/54	$<0.0001$

Appendix

Table 11: Transfer-model outcome comparisons on the frozen 169-task SWE-bench Verified pressure cohort at a 20,480-token window and the fixed 480-second attempt endpoint. F2PF differences are treatment minus control in percentage points. Paired task bootstraps give 95% intervals in the same units. Exact paired sign tests compare F2PF directions, while exact paired McNemar tests compare resolution as a secondary outcome. The Holm column applies to the pre-registered Devstral–Nemotron family, whereas the separately registered Qwen3.8 comparison is outside that family.

benchmark	repositories	F2PF difference [95% interval]	higher T:C:tie ( $p_{\mathrm{Holm}}$ )
SWE-bench Verified	11	21.1 [13.7, 25.0]	7:0:4 ( $0.0469$ )
SWE-bench Pro	11	17.5 [11.8, 24.3]	11:0:0 ( $0.00586$ )
FeatureBench	22	9.1 [5.8, 14.1]	16:2:4 ( $0.00586$ )

Appendix

Table 12: Repository-cluster sensitivity of mean per-task F2PF in the three primary Qwen3.6 pressure comparisons. Differences are treatment minus control in percentage points. The final column gives repository-level direction and the Holm-adjusted exact sign-test

p

-value. The adjustment covers the six F2PF and resolution tests.

benchmark	repositories	resolution difference [95% interval]	higher T:C:tie ( $p_{\mathrm{Holm}}$ )
SWE-bench Verified	11	17.2 [9.6, 20.1]	6:0:5 ( $0.0625$ )
SWE-bench Pro	11	13.0 [8.2, 19.1]	11:0:0 ( $0.00586$ )
FeatureBench	22	0.5 [0.0, 1.5]	1:0:21 ( $1$ )

Appendix

Table 13: Repository-cluster sensitivity of complete resolution in the same comparisons. Differences are treatment minus control in percentage points. The final column uses the same six-test Holm family as Table 12 .

item	specification
Fit	We fit a logistic regression of stopping while reading on $\log_{2}(\text{demand}/\text{window})$ .
Interval	We use the 2.5th to 97.5th percentile of the bootstrap distribution.
Resampling	We draw 2,000 repository samples with replacement, using seed 20260806.
Excluded draws	We exclude draws with one outcome class, a singular fit, or a non-positive fitted slope.
Paired ratio	We use the same repository sample for both arms.
Valid ratios	All 2,000 paired samples produced estimates on all three benchmarks.

Appendix

Table 14: Reading-boundary fit and uncertainty calculation.

benchmark	$n$	control location	treatment location	stretch
SWE-bench Verified	166	0.71 [0.59, 0.84]	1.81 [1.43, 2.21]	2.53 [1.81, 3.59]
SWE-bench Pro	315	0.97 [0.90, 1.04]	2.15 [1.72, 3.23]	2.21 [1.80, 3.25]
FeatureBench	170	0.93 [0.82, 1.01]	2.09 [1.74, 2.34]	2.26 [1.86, 2.56]

Appendix

Table 15: Fitted Qwen3.6 reading boundaries. Each location gives the demand-to-window ratio where half the runs stop before the first source change. Brackets give repository-cluster bootstrap 95% intervals. Stretch equals the treatment location divided by the control location.

Cell	$n$	command	errors	condense	decay
Verified (20k)	169	484/117	313/105	0/0	4079/140
Verified (43k)	169	786/150	435/142	0/0	3212/107
Pro (49k)	316	6926/236	6181/230	519/104	9318/246
FeatureBench (47k)	183	1250/131	1017/127	44/16	5057/143

Appendix

Table 16: Observed context and command activity in the paired populations used for this analysis. Each entry after

n

gives the event count followed by the number of sessions with at least one event; all listed markers are zero in the corresponding control sessions.

Cell	checks	valid	setup	positive	applies	restores
Verified (20k)	2193/168	2193/168	0/0	45/31	56/31	24/21
Verified (43k)	2895/169	2895/169	0/0	50/36	62/36	32/29
Pro (49k)	10109/314	10109/314	0/0	2464/153	633/153	240/145
FeatureBench (47k)	3490/181	3490/181	0/0	424/84	262/84	90/75

Appendix

Table 17: Observed detector and conditioned-action activity in the same paired populations. Each entry gives the event count followed by the number of sessions with at least one event. The applies count excludes later restorations; all listed markers are zero in the corresponding control sessions.

Figure 9: Assembled prompt length by turn for six selected task pairs, two per benchmark. The horizontal reference is the configured window. Panel subtitles report control followed by treatment. The Verified examples are treatment-only resolutions. The Pro and FeatureBench examples have large positive paired F2PF differences.

family	model	policy setting	solutions / $n$	mean per-task F2PF (%)
R6 depth	Qwen3.6	frozen caps, prior run	6/15	45.3
		frozen caps, rerun	7/15	47.5
		all caps $\times$ 0.5	5/15	45.0
		all caps $\times$ 2.0	4/15	33.3
R7/R8 timing	Qwen3.6	activation 0.50, protect 4	3/15	22.2
		activation 0.25, protect 4	2/15	13.3

Appendix

Table 18: Exploratory single-run probes of context depth and activation timing. Rows compare only with the frozen setting for the same model and cohort.

Explore similar work

What Does a Harness Buy? Tokens, Mostly
Oct 3, 2026Yangze Liu, Zhongyi HanLLM Agent Harnesses AI Coding Agents
Shandong University
Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite
Sep 8, 2026Mohsen ArjmandiCoding Agents LLM Agent Evaluation
evolutionID GmbH, Germany
An Empirical Study of Harness Design for Coding Agents
Sep 17, 2026Run-Ze Fan, Zihao Zhang, Simin Ma +6Coding Agents Agent Harness Optimization
UMass Amherst · Emory University · UNC Charlotte

Oct 3, 2026, cs.AIJ/K move · Enter open · S save

What Does a Harness Buy? Tokens, Mostly

Yangze Liu, Zhongyi Han

Shandong University

LLM Agent Harnesses AI Coding Agents LLM Agent Evaluation LLM Agents

Open paper AskPDF arXiv

A coding agent is a language model wrapped in a harness: the system prompt, the tool set, and the context management that turn a chat model into something that can work inside a repository. Production harnesses ship releases daily, vendors advertise pass-rate gains from harness changes, and leaderboards mix harnesses freely. What is rarely measured is how much the harness itself moves the score when the model is held fixed. We run five models through three production harnesses, Claude Code, mini-SWE-agent, and OpenCode, on SWE-bench Verified, and rerun the same configurations to calibrate how much a score moves when nothing changes but the run. On 447 tasks and the two models we ran there, Claude Code and mini-SWE-agent, the heaviest and the lightest harness, are equivalent within five points. On a 45-task hard subset and five models, swapping the harness flips as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness wins in one run are not the tasks it wins in the next. The one harness effect that clears the noise is a loss, not a gain: OpenCode trails by up to 9 points on the large pool, and on one model half of that gap sits in runs its output cap cut short. What the harness does decide is the bill. With the same model, the same tasks, and one price list, cost per task differs by up to 3x across harnesses. The gap is set at the first call, by the preamble of system prompt and tool schemas each harness sends with every step, and scaled by the number of steps; per-step growth and per-call tool output differ far less. The provider's price for cached input scales the bill and does not reorder it. The rerun data also give the resolution a harness comparison needs: at the discordance we observe, 45 tasks catch a 13-point gap only half the time and no gap with 80% power, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim.

Abstract

Explore similar work

What Does a Harness Buy? Tokens, Mostly

Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite

An Empirical Study of Harness Design for Coding Agents