Design and Cross-Domain Evaluation of an Agent Execution Harness

Saurav Joshi; Manas Gupta; Ashwin Venkatesh Raman; Akshay Budhkar

cs.AIOct 18, 2026

Design and Cross-Domain Evaluation of an Agent Execution Harness

Authors: Saurav Joshi, Manas Gupta, Ashwin Venkatesh Raman, Akshay Budhkar

Organizations: Spine AI

Abstract

Completing complex tasks with language-model agents requires coordinating planning, execution, and review so the final result addresses the user's original objective. We present an agent execution harness that keeps the original objective separate from a revisable plan and an execution record. A planner assigns scoped tasks, workers produce outputs and artifacts, and a reviewer examines results against the original request before submitting or requesting additional work. Reusable domain-specific instructions guide these roles and are refined during construction using task inputs and execution trajectories, without official grading feedback. We evaluate complete harness configurations across terminal tasks, clinical responses, application building, and scientific questions. On Terminal-Bench 2.1, the harness passes 83 of 89 instances (93.26%); on HealthBench Professional, a Fable-configured system achieves a length-adjusted rubric score of 69.63 across 525 tasks. In a 53-task paired HealthBench comparison, the Fable-configured harness achieves an aggregate score 19.89 points higher than direct completion, despite a $-1.46$ -point median task-level change; mean inference cost rises from $0.29 to $2.13 per task. We combine these outcomes with latency, documented instruction revisions, and reviewer decisions. Evaluating outcomes alongside execution trajectories and resource use clarifies how coordination supports task completion, where it introduces regressions, and at what cost.

Figures & tables

Figure 1: Execution architecture and mission-state transition. The planner converts the original objective

\mathcal{C}

into scoped worker assignments. Workers append outputs and artifact references to the execution record, which review examines against the objective before requesting submission or follow-up work. Dependency edges determine dispatch order. Reviewer session initialization and benchmark-specific submission interfaces are detailed in Appendix B .

Benchmark	Native metric	Harness worker	Harness	Anthropic	OpenAI
Terminal-Bench 2.1	Task pass rate	GPT-5.6 Sol	93.26%	88.0%	91.9%
HBP	Length-adjusted rubric score	Fable 5-configured	69.63	66.0	60.5
ViBench	Plan Pass@1	Opus 4.8	93.24%	87.8%	86.5%
DrugDiscoveryBench	Full-credit pass rate	GPT-5.5	60.98%	50.04%	51.6%

Table 1: Harness evaluation results (July 2026) and published contextual references. The planner and reviewer are configured with Opus 4.8 in all four harness configurations. Anthropic references are Mythos 5 (terminal/HBP), Opus 4.8 (ViBench), and Sonnet 5 (DrugDiscoveryBench); OpenAI references are Sol Ultra, Sol, GPT-5.5, and GPT-5.5 respectively. References use their published systems and are not paired or resource-matched comparisons. The HBP harness configuration includes mixed observed worker models.

Benchmark (slice)	Native metric	Baseline Fable 5	Harness Fable-configured	Harness Opus-configured
Terminal-Bench 2.1 (9)	Task pass rate	100%	100%	100%
HBP (53)	Length-adjusted rubric score	52.94	72.83	59.68
SWE-Bench Pro (74) †	Task pass rate	81.1%	82.4%	81.1%

Table 2: Fixed-task slice outcomes on seeded, stratified 10% samples (seed 20260714 ; Section 3 ). Parentheses give task counts;

\dagger

denotes the previously reported SWE-Bench Pro slice. The baseline is execution without our harness, with direct model completions for HBP. Fable’s safeguards can route flagged requests to Opus 4.8; the HBP Fable-configured row includes both models (Section 3 ). Appendix A describes the populations.

Configuration	Direct	Harness	Mean $\Delta$	Median $\Delta$
Fable-configured	52.94	72.83	$+19.89$	$-1.46$
Opus-configured	60.74	59.68	$-1.06$	$-0.05$

Table 3: HBP comparisons on 53 common tasks. Direct and Harness give aggregate native scores; mean and median

\Delta

summarize paired task-level changes (Harness minus Direct), in score points. The Fable-configured arm includes mixed observed worker models.

Evaluation	Harness total	Harness per task	Anthropic cost ref. Fable ∗ /Opus 4.8	OpenAI cost ref. per task	Harness mean latency
Terminal-Bench 2.1, Sol (89)	$192.84	$2.17	$0.82 ∗	$5.14	16.94 min
HBP, Fable-configured (525)	$845.10	$1.61	$0.27 ∗	$0.18	6.01 min
ViBench, Opus 4.8 (24)	$251.24	$10.47	$2.04	$2.42	34.45 min
DrugDiscoveryBench, GPT-5.5 (82)	$536.28	$6.54	—	$2.80	21.79 min

Table 4: Model-inference costs in USD; reference costs are per task. Harness costs and latency cover the same evaluated instances as the corresponding performance results; costs include planning, workers, and review. Unmarked references match the configurations in Table 1 ( OpenAI, 2026b ; OpenAI, 2026c ; ViBench, 2026 ; Akyürek et al., 2026 ) . ViBench costs and latency are per application build, not per browser test plan. OpenAI’s terminal and HBP costs are estimated API costs. ∗ Anthropic cost references use Fable 5 rather than Mythos 5: terminal, 83.8%, with mean inference cost over 445 attempts ( Terminal-Bench Team, 2026b ) ; HBP, 60.9 in OpenAI’s independent evaluation. A dash denotes an unreported value.

Task type	Direct	Harness
Care-instruction translation	$-100$	100
Latest clinical guideline	100	0
Interacting conditions	100	100
Dosing despite escalation warnings	0	0

Table 5: Balanced retrospective illustrations: favorable change, regression, shared high coverage, and shared low coverage. Values are raw signed rubric scores, not binary passes or the final length-adjusted metric.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Grade-blind specialization as construction of domain instructions, followed by evaluation of configured systems. The schematic separates construction feedback from official grading. Full-benchmark evaluation includes construction inputs and the remaining benchmark tasks. Native aggregation is benchmark-specific, and reported resource totals exclude setup and infrastructure.

Benchmark	Planner	Worker	Reviewer
Terminal-Bench 2.1	xhigh	ultra	xhigh
HealthBench Professional	xhigh	xhigh	xhigh
ViBench	high	max	high
DrugDiscoveryBench	max	xhigh	max

Appendix

Table 6: Role-level reasoning-effort settings for the four primary benchmark configurations.

Role	Earlier instruction / procedure	Revised instruction / procedure
HBP planner	understand the medical task, then hand it to one worker	frame the task for one worker—do not solve it yourself
HBP reviewer	General accuracy and coverage checking	Re-derive the key clinical facts yourself from the source conversation, rather than the plan’s framing
Terminal worker	Orient before acting	Produce a complete runnable artifact at the requested path early, then iterate
Terminal reviewer	General completion checking	Programmatically test machine-checkable requirements rather than rely on visual plausibility

Appendix

Table 7: Procedural revisions recorded in the setup logs. Italicized phrases are instruction excerpts; other cells paraphrase the changes. Chronology and write success do not identify an isolated performance effect.

Panel of Figure 3 — Figure 3: ViBench Notes artifacts after a cold reload. The harness returns to a password gate with note data hidden (left), whereas the comparison artifact remains unlocked (right). This is a task-specific state observation, not the aggregate Plan Pass@1 measure or a component ablation.

Artifact property	Before repair	After repair
G-code text segment	iZ	iz
G-code output bytes	26	26
SQL final-column types	Text	Integer
SQL printable result rows	500	500

Appendix

Table 8: Recorded artifact changes following reviewer-created tasks. Both final task outcomes pass; the table compares local artifact properties rather than intermediate official grades.

Evaluation	Mean	Median	p90
HBP, Fable-configured	6.01	5.37	8.14
HBP, Kimi-configured	8.85	7.73	13.85
Terminal-Bench 2.1	16.94	13.77	31.79

Appendix

Table 9: Recorded task latency distributions, in minutes. HBP uses mission-event spans; terminal uses Harbor wall time.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 357 papers

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +10Computer-Use Agent Benchmarks AI Agent Reliability
Peking University · Qiyuan Tech
$A^2E$ : An End-to-End Agent Auditing Engine
Aug 7, 2026Haoning Wang, Mingxun Zhang, Chenyue Yu +4Tool-Use Evaluation AI Agent Auditing
Shanghai Artificial Intelligence Laboratory
From Question Answering to Task Completion: A Survey on Agent System and Harness Design
Jun 14, 2026Jianyuan Guo, Zhiwei Hao, Chengcheng Wang +14LLM Agent Evaluation LLM Agents
Department of Computer Science, City University of Hong Kong, HKSAR, China

May 27, 2026J/K move · Enter open · S save

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows

Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li+9

Peking University · Qiyuan Tech

Computer-Use Agent Benchmarks AI Agent Reliability LLM Agent Evaluation AI Agent Benchmarks LLM Agent Harnesses LLM Agent Reliability

Open paper AskPDF arXiv

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.

Abstract

LLM Agent Harnesses

Explore similar work

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows

A2EA^2EA2E : An End-to-End Agent Auditing Engine

From Question Answering to Task Completion: A Survey on Agent System and Harness Design

$A^2E$ : An End-to-End Agent Auditing Engine