Harness Engineering for Software Engineering via Modular Executable Dev-Primitives

Haibo Jin; Xinjie Li; Peng Kuang; Haohan Wang

cs.SEOct 6, 2026

Harness Engineering for Software Engineering via Modular Executable Dev-Primitives

Authors: Haibo Jin, Xinjie Li, Peng Kuang, Haohan Wang

Organizations: University of Illinois Urbana-Champaign, USA · The Pennsylvania State University, USA

Abstract

Large language models (LLMs) equipped with terminal access have demonstrated strong capabilities in automating software engineering tasks. However, existing agents remain brittle on long-horizon workflows, where they must repeatedly reconstruct program state scattered across source files, configurations, tests, dependencies, and runtime behavior, leading to increasingly long interaction histories, context explosion, and semantic drift. Large repositories further complicate the identification of task-relevant components. To address these challenges, we introduce \textbf{Dev-Primitives} (\emph{Development Primitives}), a modular and executable abstraction that transforms repository components from passive software artifacts into active participants in software engineering. Each Dev-Primitive pairs a repository artifact with a resident LLM, which gives the artifact an agent-native interface grounded in its own implementation and dependencies, enabling natural-language reasoning, inter-component communication, and localized self-modification. Building on Dev-Primitives, we propose \textbf{HERMES}, a Harness Engineering framework for software engineeRing via Modular Executable Dev-PrimitiveS, which instantiates these primitives at repository scale through a dependency-aware dynamic activation mechanism and a bug diagnosis mechanism that maps execution evidence back to the components that must be revised. Extensive experiments on four software engineering benchmarks demonstrate that HERMES outperforms matched baseline harnesses by 12.4% on average. Moreover, when paired with strong activation and diagnosis models, HERMES, even with Qwen3-8B Dev-Primitives, remains within 4.5% of the homogeneous GPT-5.6 Sol configuration across all four benchmarks, while reducing inference cost by 26.2% on Terminal-Bench 4.0, highlighting the importance of harness design in software engineering agents.

Figures & tables

Figure 1: Overview of HERMES. Dynamic activation selects the repository components relevant to the task and instantiates Dev-Primitives, which modify their own artifacts and exchange requirements in natural language. The modified repository is executed, and bug diagnosis maps the observed failure back to the components that must be revised, reactivating only those for the next round.

	Baseline Results			HERMES Results
Backbone	Resolved (%) $\uparrow$	Cost/Test ( $)$ \downarrow$	Latency (s) $\downarrow$	Resolved (%) $\uparrow$	Cost/Test ( $)$ \downarrow$	Latency (s) $\downarrow$	$\Delta$ Res. (p.p.) $\uparrow$
Harness	mini-SWE-agent			HERMES
GPT-5.6 Sol	96.20 $\pm$ 0.86	1.15	182.37	97.00 $\pm$ 0.76	1.80	205.34	+0.80
GPT-5.6 Terra	95.40 $\pm$ 0.94	0.40	180.00	96.20 $\pm$ 0.86	0.98	196.71	+0.80
GPT-5.6 Luna	93.00 $\pm$ 1.14	0.04	201.07	95.60 $\pm$ 0.92	0.10	218.46	+2.60
Claude Fable 5	95.00 $\pm$ 0.98	2.05	356.18	96.00 $\pm$ 0.88	4.50	381.27	+1.00

Table 1: Performance and efficiency on SWE-bench Verified. Resolved rates are mean

\pm

standard error over 500 instances.

		Baseline Results		HERMES Results
Model	Effort	Comp. (%) $\uparrow$	Cost ( $)$ \downarrow$	Comp. (%) $\uparrow$	Cost ( $)$ \downarrow$	$\Delta$ Comp. (p.p.) $\uparrow$
Harness		Claude Code		HERMES
Claude Opus 5	medium	28.5	38.4	36.5	58.6	+8.0
	high	34.5	55.7	41.5	82.4	+7.0
Claude Sonnet 5	medium	15.0	11.9	22.0	22.7	+7.0
	high	6.0	24.6	25.5	34.8	+19.5

Table 2: Performance and cost on SWE Refactor Bench.

		Baseline Results			HERMES Results
Model	Effort	Resolution (%) $\uparrow$	Tokens $\downarrow$	Cost ( $)$ \downarrow$	Resolution (%) $\uparrow$	Tokens $\downarrow$	Cost ( $)$ \downarrow$	$\Delta$ Res. (p.p.) $\uparrow$
Harness		Codex			HERMES
GPT-5.6 Sol	medium	33.0 $\pm$ 3.4	3.6B	1.9k	51.8 $\pm$ 2.2	6.8B	3.4k	+18.8
	max	37.3 $\pm$ 3.8	4.4B	2.5k	55.5 $\pm$ 3.1	8.9B	4.7k	+18.2
GPT-5.6 Terra	medium	18.2 $\pm$ 3.0	4.4B	1.2k	44.5 $\pm$ 3.3	7.2B	2.2k	+26.3
	max	21.5 $\pm$ 3.3	5.7B	1.7k	48.2 $\pm$ 2.5	9.1B	2.9k	+26.7

Table 3: Performance and efficiency on Terminal-Bench 4.0. Resolution rates are mean

\pm

standard deviation over 5 runs; tokens and costs are totals over the full evaluation.

	Baseline Results					HERMES Results
Model	Build $\uparrow$	Monitor. $\uparrow$	Issue $\uparrow$	Test $\uparrow$	Avg. $\uparrow$	Build $\uparrow$	Monitor. $\uparrow$	Issue $\uparrow$	Test $\uparrow$	Avg. $\uparrow$	$\Delta$ Avg. (p.p.) $\uparrow$
Harness	Codex					HERMES
GPT-5.6 Sol	77.78	38.24	40.97	42.58	49.89	83.33	44.12	46.13	48.06	55.41	+5.52
GPT-5.6 Terra	74.07	35.29	38.71	40.32	47.10	79.63	41.18	43.55	44.84	52.30	+5.20
Harness	Claude Code					HERMES
Claude Opus 5	79.63	41.18	42.90	44.52	52.06	85.19	47.06	47.10	49.03	57.10	+5.04

Table 4: Performance on DevOps-Gym across four software engineering stages. DeepSeek-V4 uses max reasoning effort.

Benchmark	Method	Score (%) $\uparrow$
SWE-bench	MAGIS	13.9
SWE-bench	HERMES	36.6
NL2Repo-Bench	CodeTeam (PE)	34.6
	CodeTeam (SFT)	42.3
	HERMES	44.9

Table 5: Reference comparison with related repository-level agent frameworks.

			SWE-bench	SWE Refactor	Terminal-Bench	DevOps-Gym
Activation	Dev-Primitives	Diagnosis	Resolved (%) $\uparrow$	Composite (%) $\uparrow$	Resolved (%) $\uparrow$	Avg. (%) $\uparrow$
Homogeneous Backbones
Qwen3-8B	Qwen3-8B	Qwen3-8B	80.6	10.5	27.6	37.48
GPT-5.6 Luna	GPT-5.6 Luna	GPT-5.6 Luna	95.6	16.0	29.1	46.64
DeepSeek-V4	DeepSeek-V4	DeepSeek-V4	82.8	15.0	39.4	53.08
GPT-5.6 Sol	GPT-5.6 Sol	GPT-5.6 Sol	97.0	31.0	51.8	55.41

Table 6: Effect of backbone assignment across HERMES components. The lower blocks keep Dev-Primitives at Qwen3-8B and scale activation and/or diagnosis; parentheses denote improvement over the homogeneous Qwen3-8B configuration.

Configuration	Resolved (%)	Cost ($)
Qwen3-8B / Qwen3-8B / Qwen3-8B	27.6	0.70k
Qwen3-8B / Qwen3-8B / GPT-5.5	46.1	1.56k
GPT-5.6 Sol / Qwen3-8B / GPT-5.5	49.7	2.13k
GPT-5.6 Sol / Qwen3-8B / GPT-5.6 Sol	50.6	2.51k
GPT-5.6 Sol / GPT-5.6 Sol / GPT-5.6 Sol	51.8	3.40k

Table 7: Cost–performance trade-off on Terminal-Bench 4.0. Configurations are ordered as Activation / Dev-Primitives / Diagnosis.

Benchmark	Recall	Precision	Avg. Act.	Solved R.	Failed R.
SWE-bench Verified	95.4	72.6	4.7	96.1	71.4
SWE Refactor Bench	83.3	68.4	12.6	94.7	78.2
Terminal-Bench 4.0	84.5	64.9	6.8	93.5	74.8
DevOps-Gym	85.7	70.3	5.4	95.0	76.6

Table 8: Dev-Primitive selection quality with GPT-5.6 Sol. DevOps-Gym excludes monitoring tasks.

Setting	SWE-bench	Refactor	Terminal	DevOps
HERMES	97.0	31.0	51.8	55.41
w/o Inter-Primitive Comm.	94.2 ( $\downarrow$ 2.8)	23.5 ( $\downarrow$ 7.5)	46.1 ( $\downarrow$ 5.7)	50.34 ( $\downarrow$ 5.07)
w/o On-Demand Activation	95.8 ( $\downarrow$ 1.2)	28.0 ( $\downarrow$ 3.0)	49.4 ( $\downarrow$ 2.4)	53.26 ( $\downarrow$ 2.15)
w/o Execution Feedback	93.8 ( $\downarrow$ 3.2)	21.5 ( $\downarrow$ 9.5)	43.6 ( $\downarrow$ 8.2)	48.17 ( $\downarrow$ 7.24)
w/o Diagnosis Feedback	92.6 ( $\downarrow$ 4.4)	20.0 ( $\downarrow$ 11.0)	41.2 ( $\downarrow$ 10.6)	46.73 ( $\downarrow$ 8.68)

Table 9: Component ablation with GPT-5.6 Sol. Metrics are resolved rate for SWE-bench and Terminal-Bench, composite score for SWE Refactor, and average score for DevOps-Gym. Other backbones follow the same ordering (Appendix F.2 ); setting definitions are in Appendix F.6 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	SWE-bench	Refactor	T-Bench	DevOps	Scaling	Ablation
OpenAI
GPT-5.6 Sol	✓	✓	✓	✓	✓	✓
GPT-5.6 Terra	✓	✓	✓	✓
GPT-5.6 Luna	✓	✓	✓	✓	✓
GPT-5.5	✓				✓
GPT-5.4, GPT-5.4 Mini, GPT-5 Mini	✓

Appendix

Table 10: Backbone models used in our experiments. Bold denotes primary backbones. Scaling covers Tables 6 – 7 , and Ablation covers Table 9 . GPT-5.6 Sol is additionally used in Tables 8 and 12 . Reasoning effort follows the setting reported in each table.

		SWE-bench	SWE Refactor	Terminal-Bench	DevOps-Gym
Backbone	Setting	Resolved (%) $\uparrow$	Composite (%) $\uparrow$	Resolved (%) $\uparrow$	Avg. (%) $\uparrow$
GPT-5.6 Sol
GPT-5.6 Sol	HERMES	97.0	31.0	51.8	55.41
GPT-5.6 Sol	w/o Inter-Primitive Communication	94.2 ( $\downarrow$ 2.8)	23.5 ( $\downarrow$ 7.5)	46.1 ( $\downarrow$ 5.7)	50.34 ( $\downarrow$ 5.07)
GPT-5.6 Sol	w/o On-Demand Activation	95.8 ( $\downarrow$ 1.2)	28.0 ( $\downarrow$ 3.0)	49.4 ( $\downarrow$ 2.4)	53.26 ( $\downarrow$ 2.15)
GPT-5.6 Sol	w/o Execution Feedback	93.8 ( $\downarrow$ 3.2)	21.5 ( $\downarrow$ 9.5)	43.6 ( $\downarrow$ 8.2)	48.17 ( $\downarrow$ 7.24)

Appendix

Table 11: Complete component ablation of HERMES across three backbone models. Parentheses denote absolute changes relative to complete HERMES with the same backbone.

$B$	SWE-bench	SWE Refactor	Terminal-Bench	DevOps-Gym
0	92.2 ( $\downarrow$ 4.8)	18.5 ( $\downarrow$ 12.5)	39.7 ( $\downarrow$ 12.1)	45.91 ( $\downarrow$ 9.50)
1	94.6 ( $\downarrow$ 2.4)	24.0 ( $\downarrow$ 7.0)	45.2 ( $\downarrow$ 6.6)	50.17 ( $\downarrow$ 5.24)
2	96.4 ( $\downarrow$ 0.6)	28.5 ( $\downarrow$ 2.5)	48.8 ( $\downarrow$ 3.0)	53.72 ( $\downarrow$ 1.69)
3	97.0	31.0	51.8	55.41
4	97.2 ( $\uparrow$ 0.2)	31.5 ( $\uparrow$ 0.5)	52.4 ( $\uparrow$ 0.6)	55.79 ( $\uparrow$ 0.38)
5	97.2 ( $\uparrow$ 0.2)	31.5 ( $\uparrow$ 0.5)	52.7 ( $\uparrow$ 0.9)	55.88 ( $\uparrow$ 0.47)

Appendix

Table 12: Effect of the revision budget

B

, with changes measured relative to the default

B=3

SWE-bench Verified

SWE Refactor Bench

Terminal-Bench 4.0

DevOps-Gym

Setting

Res.

\uparrow

Tok.

Cost

Comp.

\uparrow

Tok.

Cost

Res.

\uparrow

Tok.

Cost

Avg.

\uparrow

Tok.

Cost

HERMES

97.0

1.00

\times

1.00

\times

31.0

1.00

\times

1.00

\times

51.8

1.00

\times

1.00

\times

55.41

1.00

\times

1.00

\times

Single editor

93.4

0.86

\times

0.85

\times

16.8

0.90

\times

0.89

\times

38.8

0.86

\times

0.84

\times

44.21

0.87

\times

0.86

\times

\Delta

-

3.6

-

14.2

-

13.0

-

11.20

Appendix

Table 13: Controlled comparison between HERMES and a centralized single-editor variant. Both use the same activation-selected component set, execution environment, diagnosis module, GPT-5.6 Sol backbone with medium reasoning effort, and

B=3

. Tok. and Cost are normalized to HERMES on each benchmark.

Setting	Resolution (%) $\uparrow$	Tokens	Cost ($)
GPT-5.6 Sol
Single editor	38.8	5.9B	2.90k
Single editor + compute matching	45.5	6.7B	3.35k
HERMES	51.8	6.8B	3.40k

Appendix

Table 14: Compute and context controls on Terminal-Bench 4.0. All settings use the same activation-selected component set, execution environment, diagnosis module, and

B=3

; GPT-5.6 Sol uses medium reasoning effort.

	Avg. Primitive Calls
Benchmark	HERMES	w/o Activation	Relative Cost w/o Activation
SWE-bench Verified	9.6	28.7	1.61 $\times$
SWE Refactor Bench	25.1	64.3	1.88 $\times$
Terminal-Bench 4.0	14.2	39.6	1.73 $\times$
DevOps-Gym	11.7	33.2	1.67 $\times$

Appendix

Table 15: Effect of on-demand activation. Primitive calls include repeated invocations during communication and revision; cost is relative to HERMES.

Component	Action	Initial local objective
db/models/fields/json.py	modify	Preserve non-ASCII characters when preparing JSONField values
forms/fields.py	modify	Update prepare_value to preserve non-ASCII characters
forms/utils.py	modify	Avoid ASCII escaping in JSON-formatted form output
core/serializers/json.py	modify	Preserve non-ASCII characters in core JSON serialization
forms/widgets.py	inspect	Inspect the rendering path and communicate relevant constraints
utils/encoding.py	inspect	Inspect shared encoding behavior and communicate Unicode-related constraints

Appendix

Table 16: Activated Dev-Primitives and their initial component-local responsibilities. Activation does not require a component to be modified.

From	To	Message (abridged)	Local consequence
forms/widgets.py	forms/fields.py	Use ensure_ascii=False when preparing JSON values so Unicode is preserved during rendering.	Refine prepare_value
db/models/fields/json.py	core/serializers/json.py	Keep JSON encoding behavior consistent across model-field and serializer paths.	Refine serializer behavior
utils/encoding.py	core/serializers/json.py	Preserve Unicode rather than emitting ASCII escape sequences during serialization.	Reinforce serializer objective

Appendix

Table 17: Representative directed messages from the collaboration round. Messages are abridged for space.

Explore similar work

May 13, 2026cs.SE

AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents

Foundation models have transformed automated code generation, yet autonomous software-engineering agents remain unreliable in realistic development settings. The dominant explanation locates this gap in model capability. We propose a different locus: software-engineering capability emerges from a model-harness-environment system, in which a runtime substrate -- the harness -- mediates how a foundation-model agent observes a project, acts on it, receives feedback, and establishes that a change is complete. We formalize this substrate as an AI Harness Engineering and identify eleven component responsibilities: task specification, context selection, tool access, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing, and intervention recording. We operationalize the harness through a four-level ladder (H0-H3) that progressively exposes runtime support to the agent, and we propose a trace-based evaluation protocol that converts each agent run into an auditable episode package. Applied to a controlled validation task, the framework yields episode packages whose evidence structure varies systematically with harness level: lower levels produce only a final patch, higher levels produce reproduction logs, failure attributions, deterministic requirement checks, and structured verification reports. The framework reframes the central question of autonomous software engineering from whether a foundation model can produce a patch to whether the model-harness-environment system can produce a verifiably correct, attributed, and maintainable change. We outline a research program for the runtime systems that foundation-model software agents will require.

Hailin Zhong, Shengxin Zhu

Hong Kong Baptist University, Hong Kong, China · Beijin Normal University, Zhuhai, China

Agent Harness Software Engineering Runtime

Ask PDF

Sep 1, 2026cs.SE

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16

Agent Harness

Ask PDF

Oct 6, 2026cs.SE

Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete repositories remains difficult because interacting modules, interfaces, configurations, tests, and dependencies must work together. We introduce Code Primitives, agent-native reusable executable components with interface contracts, dependency closures, validation tests, and provenance. Each primitive uses a resident LLM to assess relevance and adapt its implementation, interfaces, and dependencies to the target repository, and we organize 1,424 validated primitives in CodeFace, a searchable library for repository construction. We introduce LEGO (Large-scale repository Engineering via aGent-native reusable cOde primitives), which activates task-relevant primitives, integrates their adapted implementations with task-specific code while resolving cross-component constraints, and revises the result against executed tests. To measure construction end to end, we build LEGO-REPO, a benchmark of 522 executable reconstruction tasks spanning seven software domains, 22 capability tracks, and five difficulty levels, scored against native test suites between an empty-package floor and original-source ceiling. The strongest of 13 evaluated backbones reaches a delivery score of 0.318 and scores zero on 41.0% of tasks; LEGO improves all 13 by 0.1474 on average and raises GPT-5.6-terra from 0.3180 to 0.5134 (+61.4%). In controlled comparisons, adapted primitives outperform retrieved code supplied as context or vendored unchanged. The effect persists against independent repository agents, across three external benchmarks, and with a disjointly re-mined CodeFace; GPT-OSS-20B for adaptation and diagnosis retains 95.1% of the homogeneous score at 24.0% lower cost.

Haibo Jin, Peng Kuang, Xucheng Yu +3

University of Illinois Urbana-Champaign, USA

Topic pending

Ask PDF