When Agents Fail: Model or Harness--Who's to Blame?

Shao Zhang; Hangfan Zhang; Yuanyuan Shi; Hao Li; Min-Le Su; Kangcong Li; Jingping Wang; Chunjiang Mu

cs.AIOct 23, 2026

When Agents Fail: Model or Harness--Who's to Blame?

Authors: Shao Zhang, Hangfan Zhang, Yuanyuan Shi, Hao Li, Min-Le Su, Kangcong Li, Jingping Wang, Chunjiang Mu,

Organizations: Shanghai Artificial Intelligence Laboratory

Abstract

LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Completion Rate (VCR), how reliably agents finish normally and deliver evaluable results, irrespective of task correctness; (ii) Model-Harness Fit, how well a harness supports task achievement through valid completion for a given model; (iii) Cross-Model Fit, a harness's overall suitability across evaluated models; (iv) Interaction Residuals, pairing-specific execution strengths and weaknesses hidden by component-level summaries; and (v) Token Load, the token demands of agent execution, including failed attempts. We evaluate four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, and DeepAgents) with four core models, plus two fixed native model-harness systems, on six benchmarks covering context management, tool use, long-context planning, multimodal interaction, and artifact handoff. Including native references and stress conditions, 111 evaluation settings yield 52,168 trajectories. Harness rankings vary across models and benchmarks; task performance and valid completion can favor different pairings, and similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses on the shared four-benchmark panel. These findings support evaluating model-harness interactions, training transferable harness-use capabilities in models, and harness designs adapted to the model and task.

Figures & tables

Figure 1: Overview of the model–harness evaluation framework.

Figure 2: Model–harness composition and execution loop. The harness prepares model context and executes approved actions; model outputs $y_{t}$ and executed actions $a_{t}$ are distinct.

Benchmark	Tasks	Key interaction demands
SWE-bench-45 ( Jimenez et al., 2024 ; Chowdhury et al., 2024 )	45	Repository editing, testing, patch handoff
Terminal-Bench 2 ( Merrill et al., 2026 )	89	Stateful terminal interaction
VitaBench ( He et al., 2026 )	400	Multi-turn services, domain APIs
LOCA-Bench ( Zeng et al., 2026 )	75	Long-context planning, exact-state completion
ClawMark ( Meng et al., 2026 )	100	Multi-day workflows, state changes, multimodal inputs
AgentSkillOS ( Li et al., 2026 )	30	Skill selection, artifact handoff

Table 1: Benchmark coverage and harness-sensitive interaction demands.

Model	Harness	SWE- bench-45		Terminal- Bench 2		VitaBench		LOCA-Bench		ClawMark		AgentSkill OS
Model	Harness	Best	Mean	Best	Mean	Best	Mean	Best	Mean	Best	Mean	AgentSkill OS
DeepSeek V4 Flash Preview	DeepAgents	40.00	31.11	61.80	45.79	69.00	38.00	40.00	20.67	–	–	–
	OpenCode	42.22	32.22	53.93	40.45	71.50	45.38	53.33	31.00	–	–	–
	Pi CLI	46.67	35.56	61.80	44.66	75.75	48.94	42.67	29.00	–	–	–
	Codex CLI	46.67	36.11	50.56	37.64	77.00	49.06	76.00	50.67	–	–	–
MiniMax M3	DeepAgents	55.56	31.11	46.07	21.63	77.25	51.00	33.33	24.00	75.06	54.78	80.07

Table 2: Main outcomes (0–100; scoring in Section A.3 ). Best/Mean: Best-of-4/Mean-of-4; ClawMark: continuous scores; AgentSkillOS: Task Score ( skill_200 ). Bold/underlined: best/second-best harness per core model and metric, including ties. Fixed references are excluded from highlighting and crossed analyses. Dashes: unevaluated; both DeepSeek models lack multimodal support for ClawMark and AgentSkillOS.

Figure 3: Task performance and operational validity on the balanced six-benchmark panel, summarized by Outcome Win Rate and VCR Win Rate. Each point represents a model–harness pairing; horizontal and vertical whiskers give the respective pointwise 95% task-cluster bootstrap intervals. Dashed lines: 50% (neutral).

Figure 4: Additive VCR interaction residuals (pp) for the balanced three-model panel. $*$ marks pointwise 95% task-cluster bootstrap intervals excluding zero. Benchmark cells report benchmark-specific point estimates.

Figure 5: Model–harness Fit and Cross-Model Fit on the balanced six-benchmark panel (0–100). Whiskers: pointwise 95% paired task-cluster bootstrap intervals. Higher values retain more of the best observed joint score.

Harness	MiniMax M3	Kimi K2.7	Qwen3.6	Cross-model
OpenCode	86.86 [85.11, 88.50]	87.06 [83.02, 90.94]	88.55 [80.84, 93.36]	87.48 [84.06, 89.99]
Pi CLI	73.73 [69.87, 77.73]	76.16 [71.99, 80.27]	81.70 [74.88, 84.98]	77.13 [73.49, 79.62]
Codex CLI	72.37 [68.59, 76.61]	60.45 [56.84, 63.91]	91.21 [85.55, 93.44]	73.62 [70.54, 75.82]
DeepAgents	60.56 [56.35, 64.62]	61.29 [57.11, 65.48]	71.29 [63.99, 76.22]	64.20 [60.57, 66.94]

Table 3: Fit on the balanced six-benchmark panel (0–100). Brackets give pointwise 95% paired task-cluster bootstrap intervals.

Figure 6: Joint outcome–validity shares for all 64 configurations (

n

: attempts per pairing). Bars sum to 100%; NCS gives non-clean success (%). Four repetitions per task; fixed references are excluded.

Figure 7: Clean/non-clean contributions summing to Mean Score. Right column: VCR (%). ClawMark: 400 attempts per configuration; AgentSkillOS ( skill_200 ): 120. Four repetitions per task; fixed references are excluded.

Figure 8: Exclusive execution stages on the balanced three-model panel: clean completion, pre-normal loss, and post-normal handoff loss sum to 100%. Panels report means by (a) harness and (b) benchmark. Fixed references are excluded.

Harness	Score Mean	VCR	Affected attempts % [95% CI]	Events per affected attempt
Codex CLI	11.96	100.00	6.75 [4.50, 9.25]	1.00
DeepAgents	58.65	53.25	0.00 [0.00, 0.00]	–
OpenCode	70.36	88.00	0.00 [0.00, 0.00]	–
Pi CLI	11.93	99.25	40.50 [36.25, 44.75]	1.32

Table 4: Strict Kimi K2.7 action-realization audit on ClawMark. Each harness has 400 attempts. Brackets show pointwise 95% task-cluster bootstrap intervals. For no-event cells, [0,0] reflects zero empirical bootstrap variation within the observed sample. Dashes denote undefined ratios.

	Win rate (%)		Fit variants (0–100)
Harness	Outcome	VCR	Outcome-only	Arithmetic	Cross-model
OpenCode	72.22	73.15	87.84	87.49	87.48
Pi CLI	49.07	58.33	78.20	77.20	77.13
Codex CLI	43.52	45.37	74.56	74.68	73.62
DeepAgents	35.19	23.15	68.15	64.38	64.20

Table 5: Comparison metrics and Fit ablations on the common three-model panel. Outcome-only replaces

US

with

S

; arithmetic replaces the cross-model geometric mean. Aggregation details: Section A.4 .

Omitted		DA	OC	Pi	CX	Order
Model	MiniMax	66.10	87.80	78.88	74.26	$OC>Pi>CX>DA$
Model	Kimi	65.71	87.70	77.61	81.25	$OC>CX>Pi>DA$
Model	Qwen	60.92	86.96	74.94	66.14	$OC>Pi>CX>DA$
Benchmark	SWE-45	68.86	86.06	75.45	77.75	$OC>CX>Pi>DA$
Benchmark	TB2	63.29	86.29	73.66	71.91	$OC>Pi>CX>DA$
Benchmark	Vita	57.82	85.26	73.40	70.69	$OC>Pi>CX>DA$

Table 6: Cross-model Fit after omitting one model or benchmark. Calculation details: Section A.4 . DA: DeepAgents; OC: OpenCode; Pi: Pi CLI; CX: Codex CLI.

Held-out model	Selected Fit	Best harness	Best Fit	Regret
MiniMax M3	86.86	OpenCode	86.86	0.00
Kimi K2.7	87.06	OpenCode	87.06	0.00
Qwen3.6	88.55	Codex CLI	91.21	2.66

Table 7: Leave-one-model-out selection diagnostic. Fit, outcome-only Fit, Outcome Win Rate, and VCR Win Rate all select OpenCode using the two remaining models. Regret is the difference, in Fit points, between the held-out model’s best single-harness Fit and the selected Fit.

Harness	Token-use win rate	Outcome-preserving win rate	Pareto-frontier share	Relative load
DeepAgents	51.85 [44.44, 53.70]	14.81 [11.11, 22.22]	50.00 [27.78, 55.56]	1.095 [1.043, 1.159]
OpenCode	44.44 [40.74, 51.85]	35.19 [31.48, 44.44]	61.11 [50.00, 72.22]	1.150 [1.112, 1.202]
Pi CLI	55.56 [50.00, 61.11]	24.07 [20.37, 33.33]	50.00 [38.89, 66.67]	0.906 [0.827, 0.967]
Codex CLI	48.15 [44.44, 51.85]	24.07 [16.67, 27.78]	55.56 [44.44, 61.11]	0.877 [0.838, 0.927]

Table 8: Matched resource comparisons (18 complete primary model–benchmark blocks; 72 cells). First three columns: %; brackets: pointwise 95% bootstrap intervals; bold/underlined: best/second-best. Definitions and resampling: Section 4 .

Figure 9: Repeated-run gaps and resource tails. (a) Best-of-4 minus Mean-of-4 for 76 core crossed configurations; thick segments: interquartile ranges; filled points: medians. (b) Mean versus median Token Load for 90 fully covered primary cells with both statistics; dashed line: equality. The three most skewed cells are highlighted.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Harness	By model			Head-to-head				Overall win rate
Harness	MiniMax M3	Kimi K2.7	Qwen3.6 35B-A3B	vs. DA	vs. OC	vs. Pi	vs. Codex	Overall win rate
DeepAgents	38.89	38.89	27.78	–	22.22	33.33	50.00	35.19
OpenCode	83.33	72.22	61.11	77.78	–	66.67	72.22	72.22
Pi CLI	44.44	55.56	47.22	66.67	33.33	–	47.22	49.07
Codex CLI	33.33	33.33	63.89	50.00	27.78	52.78	–	43.52

Appendix

Table 9: Six-benchmark Outcome Win Rates (%; Table 2 ). Columns: model-conditioned rates, head-to-head rates pooled across models, and the three-model mean. DA/OC: DeepAgents/OpenCode; ties receive half credit. Rates report tie-aware wins across the specified configuration comparisons.

Harness	MiniMax M3		Kimi K2.7		Qwen3.6-35B-A3B
	VCR win rate	Mean VCR	VCR win rate	Mean VCR	VCR win rate	Mean VCR
DeepAgents	25.00	60.60	27.78	59.69	16.67	79.31
OpenCode	86.11	91.68	55.56	83.09	77.78	90.70
Pi CLI	47.22	72.96	77.78	87.94	50.00	82.13
Codex CLI	41.67	71.88	38.89	78.26	55.56	89.48

Appendix

Table 10: Six-benchmark model-conditioned VCR Win Rate and Mean VCR (%). VCR Win Rate uses tie-aware comparisons; Mean VCR weights benchmarks equally. Bold/underlined: best/second-best harness within each column.

Benchmark	$\|\mathcal{C}_{b}\|$	Native outcome	Outcome W/T/L	Outcome WR	Native VCR	VCR W/T/L	VCR WR	Joint
GPT-5.5–Codex
SWE-bench-45	16	65.56	16/0/0	100.00	99.44	13/3/0	90.63	100.00
Terminal-Bench 2	16	60.39	16/0/0	100.00	72.47	10/0/6	62.50	62.50
VitaBench	16	46.56	11/0/5	68.75	100.00	4/12/0	62.50	68.75
LOCA-Bench	16	76.00	15/0/1	93.75	96.67	8/0/8	50.00	50.00
ClawMark	12	74.31	12/0/0	100.00	92.25	7/0/5	58.33	58.33

Appendix

Table 11: Fixed references versus evaluated core configurations on matched tasks. Outcome: Mean-of-4 (continuous for ClawMark) or AgentSkillOS Task Score. W/T/L: configuration-level wins/ties/losses; WR: tie-aware reference win rate; Joint: percentage of configurations Pareto-dominated in both outcome and VCR. Summaries weight benchmarks equally: six for GPT-5.5–Codex; SWE-bench-45, Terminal-Bench 2, VitaBench, and LOCA-Bench for DeepSeek Harness.

Harness	Lower quartile	Model	Lower quartile
DeepAgents	49.16	MiniMax M3	48.60
OpenCode	81.67	Kimi K2.7	60.00
Pi CLI	64.33	Qwen3.6-35B-A3B	76.00
Codex CLI	68.82

Appendix

Table 12: Nearest-rank lower quartiles of VCR (%). Harness and model orderings match the type-7 summaries.

Model	Harness	Best	Mean
DeepSeek V4 Flash Preview	DeepAgents	40.00	31.11
	OpenCode	42.22	32.22
	Pi CLI	46.67	35.56
	Codex CLI	46.67	36.11
MiniMax M3	DeepAgents	55.56	31.11
	OpenCode	82.22	62.78

Appendix

Table 13: SWE-bench-45 Best-of-4 and Mean-of-4 success rates (%; Section A.3 ). Bold/underlined: best/second-best within each core model, including ties. Fixed references are excluded from highlighting.

Model	DeepAgents	OpenCode	Pi CLI	Codex CLI
DeepSeek V4 Flash Preview	19.76	-11.49	-14.41	6.15
MiniMax M3	-8.85	7.12	1.42	0.31
Kimi K2.7	-19.97	18.23	21.42	-19.69
Qwen3.6-35B-A3B	9.06	-13.85	-8.44	13.23

Appendix

Table 15: SWE-bench-45 additive VCR interaction residual (percentage points), centered over the four-model SWE panel.

Model	DeepAgents		OpenCode		PI		Codex
Model	Best	Mean	Best	Mean	Best	Mean	Best	Mean
Qwen3.6-35BA3B	40.4	26.4	40.4	26.1	43.8	28.7	47.2	28.7
DeepSeek	61.8	45.8	53.9	40.4	61.8	44.7	50.6	37.6
MiniMax-M3	46.1	21.6	55.1	39.6	53.9	33.1	51.7	34.6
Kimi-K2.7	44.9	30.6	53.9	39.0	59.6	43.8	48.3	31.7

Appendix

Table 16: Terminal-Bench 2 results on all 89 tasks: four models, four harnesses, and fixed references. Best/Mean: Best-of-4/Mean-of-4.

Model	Harness	Delivery		In-store		OTA		Cross		Total
Model	Harness	Best	Mean	Best	Mean	Best	Mean	Best	Mean	Best	Mean
MiniMax M3	DeepAgents	81.00	59.50	83.00	58.25	76.00	47.00	69.00	39.25	77.25	51.00
	OpenCode	83.00	59.00	84.00	56.75	76.00	49.00	59.00	32.75	75.50	49.38
	Pi CLI	86.00	59.25	87.00	58.50	77.00	49.25	59.00	30.00	77.25	49.25
	Codex CLI	85.00	53.25	82.00	48.50	67.00	35.25	76.00	43.25	77.50	45.06
Kimi K2.7	DeepAgents	75.00	51.50	78.00	52.25	69.00	47.00	42.00	21.50	66.00	43.06

Appendix

Table 17: VitaBench Best-of-4/Mean-of-4 success rates (%) under strict tool permissions: 100 tasks and four independent trials (400 attempts) per environment ( Section A.3 ). Cross combines Delivery, In-store, and OTA; Total averages the four environments equally. Bold/underlined: best/second-best per core model and metric, including ties. Fixed references are excluded from highlighting.

Model	DeepAgents		OpenCode		Pi CLI		Codex CLI
Model	Best	Mean	Best	Mean	Best	Mean	Best	Mean
DeepSeek V4 Flash Preview	40.00	20.67	53.33	31.00	42.67	29.00	76.00	50.67
Qwen 3.6–35B–A3B	14.67	8.67	34.67	21.00	28.00	15.00	70.67	36.33
MiniMax M3	33.33	24.00	32.00	21.00	21.33	12.67	93.33	87.00
Kimi K2.7	17.33	8.33	42.67	27.67	50.67	34.33	86.67	54.00

Appendix

Table 19: LOCA-Bench 256K Best-of-4/Mean-of-4 success rates (%) for the four-model, four-harness panel and fixed references. Bold marks the best harness per core model and metric; references are excluded.

Model	DeepAgents	OpenCode	Pi CLI	Codex CLI
DeepSeek V4 Flash Preview	+19.71	-10.13	+10.88	-20.46
Qwen3.6-35B-A3B	+9.46	-1.04	+2.29	-10.71
MiniMax M3	-12.21	+18.63	-16.38	+9.96
Kimi K2.7	-16.96	-7.46	+3.21	+21.21

Appendix

Table 21: LOCA-Bench additive VCR interaction residual (percentage points), centered over the four-model panel.

Harness	Runtime errors	Timeout-like failures	Main observed pattern
Codex CLI	6	0	Most runs terminate normally.
Pi CLI	156	154	Timeouts, and BrokenPipeError ; some error runs still have a correct final state.
OpenCode	120	120	Timeout termination during long tasks.
DeepAgents	251	235	Long timeouts, graph recursion limits, forced completion, and partial-state recovery.

Appendix

Table 23: Runtime-error evidence for Kimi K2.7 (300 attempts per harness).

Model	Harness	Score		Full Success
		Best	Mean	Best	Mean
MiniMax M3	DeepAgents	75.06	54.78	21.00	11.00
	OpenCode	82.39	69.62	28.00	12.75
	Pi CLI	78.34	62.58	28.00	14.50
	Codex CLI	77.25	58.49	22.00	11.75
Kimi K2.7	DeepAgents	72.56	58.65	20.00	8.25

Appendix

Table 24: ClawMark Best-of-4/Mean-of-4 results (%). Score uses continuous evaluator scores; Full Success requires a run score of 1. Bold/underlined: best/second-best per core model and metric, including ties. The fixed native reference is excluded from highlighting.

Model	Harness	200 skills (primary)	900 skills (stress)
MiniMax M3	DeepAgents	80.07	83.02
	OpenCode	93.49	64.08
	Pi CLI	85.39	89.98
	Codex CLI	36.44	36.42
Kimi K2.7	DeepAgents	89.92	84.34
	OpenCode	84.94	62.87

Appendix

Table 26: AgentSkillOS mean Task Score over four runs: 200 skills (primary), 900 (matched stress). Bold/underlined: best/second-best per core model–condition block; fixed reference excluded.

Condition	Model	DeepAgents	OpenCode	Pi CLI	Codex CLI
skill_200	MiniMax M3	-2.36	+11.53	+9.31	-18.47
	Kimi K2.7	+0.76	-10.35	+9.93	-0.35
	Qwen3.6-35B-A3B	+1.60	-1.18	-19.24	+18.82
skill_900	MiniMax M3	+7.78	+6.11	+5.83	-19.72
	Kimi K2.7	-0.97	-0.14	-0.42	+1.53
	Qwen3.6-35B-A3B	-6.81	-5.97	-5.42	+18.19

Appendix

Table 28: AgentSkillOS additive VCR interaction residual (percentage points). Residuals are centered separately within each skill-pool condition.

Model	Harness	SWE- bench-45	Terminal- Bench 2	VitaBench	LOCA-Bench	ClawMark	AgentSkill OS
DeepSeek V4 Flash Preview	DeepAgents	2.358	2.484	0.933	6.181	–	–
	OpenCode	1.531	0.677	0.503	2.390	–	–
	Pi CLI	2.434	1.666	0.296	9.073	–	–
	Codex CLI	3.762	1.244	0.656	1.288	–	–
MiniMax M3	DeepAgents	6.444	1.860	0.562	6.332	6.333	2.950
	OpenCode	10.722	1.882	0.404	14.644	3.563	3.679

Appendix

Table 29: Mean primary Token Load (million tokens per scheduled attempt) for configurations with complete token accounting. Dashes denote configurations outside the evaluated panel. AgentSkillOS uses skill_200 . Fixed references are reported as complete systems, separately from cross-harness indices.

Model	Harness	Outcome	Token load	Tokens/ success	Coverage
SWE-bench-45
DeepSeek V4 Flash Preview	DeepAgents	31.11	2.358	7.579	100.00
DeepSeek V4 Flash Preview	OpenCode	32.22	1.531	4.750	100.00
DeepSeek V4 Flash Preview	Pi CLI	35.56	2.434	6.845	100.00
DeepSeek V4 Flash Preview	Codex CLI	36.11	3.762	10.419	100.00
MiniMax M3	DeepAgents	31.11	6.444	20.714	100.00

Appendix

Table 30: Resource results by benchmark, condition, model, and harness. Outcome: official Mean-of-4 or Task Score; Token Load: million tokens/attempt; Tokens/Success: million tokens. Coverage is the percentage of scheduled attempts with token accounting.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +10Computer-Use Agent Benchmarks AI Agent Reliability
Peking University · Qiyuan Tech
Stop Comparing LLM Agents Without Disclosing the Harness
May 7, 2026Yunbei Zhang, Janet Wang, Yingqiang Ge +3LLM Agent Evaluation Long-Horizon Agent Evaluation
Tulane University · Rutgers University · Independent Researcher +1
$A^2E$ : An End-to-End Agent Auditing Engine
Aug 7, 2026Haoning Wang, Mingxun Zhang, Chenyue Yu +4Tool-Use Evaluation AI Agent Auditing
Shanghai Artificial Intelligence Laboratory

May 27, 2026J/K move · Enter open · S save

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows

Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li+9

Peking University · Qiyuan Tech

Computer-Use Agent Benchmarks AI Agent Reliability LLM Agent Evaluation AI Agent Benchmarks LLM Agent Harnesses LLM Agent Reliability

Open paper AskPDF arXiv

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.

Abstract

LLM Agent Evaluation

Explore similar work

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows

Stop Comparing LLM Agents Without Disclosing the Harness

A2EA^2EA2E : An End-to-End Agent Auditing Engine

$A^2E$ : An End-to-End Agent Auditing Engine