Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?

Kidus Seyoum; Ajay Mittur

cs.SESep 24, 2026

Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?

Authors: Kidus Seyoum, Ajay Mittur

Organizations: NVIDIA

Abstract

Agent behavior depends on the harness surrounding a language model, but it remains unclear whether language models can reliably improve such harnesses for hardware-design tasks. We study automatic harness evolution around a fixed subject model on 12 proprietary design-verification root-cause localization tasks. Across five trials per task, automatically evolved harnesses increased completed attempts by 71-76% and any-hit task coverage by 80-100%, while total correct attempts improved by only 18-24%. The strongest success reproducible at least twice result improved by one task, and later candidates exchanged gains across tasks rather than preserving them. An auxiliary candidate improved on a four-task validation set excluded from search but tied its baseline on a subsequent 12-task replay containing both search and validation tasks, so the selected gain did not persist across the full pool. Across the tested lineage, useful search, evidence, and finalization behaviors appeared in different candidates but did not consistently consolidate into a single harness that dominated across tasks and metrics. In a separate CVDP cross-benchmark case study, an automatically evolved defined-width repair harness produced 35.6% more functional passes than its 142-task reference baseline; the final functional verifier scored completed outputs but was not shown to the subject agent during repair. These results support archive-aware selection when evolution yields complementary specializations without consistent consolidation.

Figures & tables

Figure 1: A candidate harness and task instantiate an isolated attempt around a fixed subject model. A separate outer loop proposes branches, verifies activation, evaluates candidates, confirms promising results, and retains both the incumbent and archive.

Evaluation track	Task type	Scoring criterion	Role in the study
Proprietary design-verification benchmark	12 root-cause localization tasks	Format-normalized function exact match	Primary within-domain diagnostic evaluation
Selected CVDP subset	12 RTL generation and repair tasks	Executable functional verifier	Selected-subset phase of the cross-benchmark case study
142-task CVDP follow-up	142 RTL generation and repair tasks	Executable functional verifier	Baseline-to-evolved comparison in the cross-benchmark case study

Table 1: Evaluation tracks and their roles in the study.

Figure 2: The primary benchmark evaluates causal diagnosis over mixed verification artifacts, while CVDP evaluates completed RTL outputs with executable functional checks. The final functional verifier was not shown to the subject agent during repair. The tracks support different claims and are not numerically interchangeable.

Harness	First trial	Success $\geq$ 2/5	Pass@5	Correct attempts	Completed
Kimi K3 baseline	2/12	5/12	5/12	17/60	34/60
Automatically evolved evidence-grounded	6/12	6/12	9/12	20/60	58/60
Automatically evolved batched-causal	5/12	5/12	10/12	21/60	60/60

Table 2: Repeated evaluation results for the baseline and selected evolved harnesses.

Figure 3: Successful trials out of five for each task and harness. Aggregate gains conceal regressions and complementary task-level strengths.

Figure 4: MiniMax task success rate and mean graded task score across 72 completed generations; the search was then stopped after no further meaningful gain. A task counts as successful when $s_{t}\geq 0.8$ ; $\bar{s}$ averages the 12 graded task scores. The two final values both round to 0.417. Promotion labels identify the principal mechanism added at each accepted step.

Stage	Principal addition	Intended effect
Evidence-first	Bounded orientation, literal search, anchored reads, forced finalization	Reduce speculative search and timeouts
Coverage ledger	Observed-text and inspected-file ledger with search/read budgets	Balance premature fixation and uncontrolled exploration
Evidence-grounded	Queries restricted to observed text; bounded sequential reads	Prevent invented queries and expensive paging
Causal commit	Multiple evidence types; source-to-propagation-to-detection ordering	Avoid naming the checker as the cause
Batched causal	Independent calls per turn plus standardized read windows	Gather more evidence within the turn budget

Table 3: Audited mechanism lineage. Later stages inherit earlier controls.

Figure 5: Best branch-attempt score by task and iteration for two evolution methods. The methods and run lengths differ, so this is a descriptive view of archive activity rather than a matched comparison.

Figure 6: Candidate archives cover more tasks than individual harnesses. In the large branching experiment, the 4/12 bar is the best archived individual and was not promoted; the promoted incumbent remained at 3/12. Archive and union bars represent portfolio coverage, not the score of a composed generic harness.

Domain-context treatment	Mean candidate pass rate	Best result	Interpretation
Static curated context	16.7%	4/12	Matched standalone control
Live knowledge retrieval	17.5%	4/12	Matched standalone control
Low-density injection	10.4%	3/12	No reliable improvement
Specificity instruction	10.3%	3/12	No demonstrated improvement
Heavy domain context	10.6%	2/12	Content changed; score stayed flat

Table 4: Domain-context treatments and best observed benchmark results.

Figure 7: Functional-pass progression on the selected 12-task CVDP subset. The candidate harnesses were automatically evolved; later candidates use explicit routing and previously scored task artifacts, so the 10/12 endpoint is not a generic-harness result. Functional passes were measured after candidate outputs were complete.

CVDP harness condition	Subject LLM	Functional passes
Reference baseline: verification loop	GLM-5.2	45/142 (31.7%)
Automatically evolved: defined-width repair	GLM-5.2	61/142 (43.0%)

Table 5: Functional-verifier results for the reference baseline and automatically evolved repair harness on the 142-task CVDP follow-up.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Evidence acquisition, coverage tracking, causal analysis, and final commitment within one subject attempt.

Figure 9: Candidate branches pass through materialization, static validation, activation checks, evaluation, and experiment-specific retention.

Design-verification task subfamily	Tasks	Typical failure or reasoning pattern
Local testbench or resource accounting	T1, T2, T3, T4, T5, T7	Guards, counters, resource accounting, and tracing a local symptom to its producer
RTL or interface-state update	T8, T12	Assignment semantics, state transitions, and calling-logic behavior
Simulation, launch, or cross-layer behavior	T6, T9, T10, T11	Stimulus, checking, memory modes, restore behavior, and multi-layer failures

Appendix

Table 6: Provisional design-verification task subfamilies represented in the benchmark.

Treatment	Knowledge treatment	Mean candidate pass rate	Best result
Static curated DV context	Documentation in proposer context	16.7%	4/12
Live DV retrieval	Proposer could retrieve specialized knowledge	17.5%	4/12
Low-density injection	About 11 terms per 1,000 words	10.4%	3/12
Specificity instruction	About 18 terms per 1,000 words	10.3%	3/12
Heavy DV context	About 25 terms per 1,000 words	10.6%	2/12

Appendix

Table 7: Domain-context treatments and their best observed scores.

Configuration	Evaluated configurations	Mean pass rate	Best score	Input tokens	Agent time
Empty-policy baselines	4	14.6%	3/12	5.3–6.7M	about 1,250 s
Evolved domain scaffolds	58	n.r.	3/12	7.4–8.2M	about 1,600 s

Appendix

Table 8: Empty-policy baselines in the incremental domain-context experiment.

Model	Passed attempts / 36	Timeouts
GLM 5.2	12/36	6
Kimi K2.6	6/36	17
Nemotron Ultra	4/36	4
MiniMax	2/36	23

Appendix

Table 9: Cross-model descriptive baseline results.

Explore similar work

Aug 3, 2026cs.LG

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54% to 66% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.

Luan Zhang, Ruochen Zhou, Dandan Song +9

Beijing Institute of Technology, China · City University of Hong Kong, China · Independent, China

Evolving Harness Agent Harness Evolutionary Optimization Methods

Ask PDF

Jul 14, 2026cs.AI

Rethinking the Evaluation of Harness Evolution for Agents

We revisit the evaluation of automatic harness evolution for LLM agents. Existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark. This protocol raises two fundamental concerns. First, harness evolution is itself an iterative search procedure that repeatedly evaluates and revises candidate harnesses using task feedback. As in agentic test-time scaling, it should therefore be compared with simple task-level search baselines under matched feedback and inference budgets to determine whether its gains arise from improved harness design or from additional search alone. Second, because the search and the final evaluation share the same benchmark, the reported gains risk overfitting to that specific task set. To address these concerns, we conduct an extensive evaluation comparing harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets, and also evaluate evolved harnesses on held-out tasks to assess whether the discovered improvements generalize. Experiments on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6 show that automatic harness evolution does not consistently outperform simple test-time scaling methods and exhibits limited generalization. Our results raise important questions about the effectiveness of automatic harness evolution and highlight the need for fairer evaluation protocols and benchmarks for automatic harness design. Our code is available at https://github.com/rethinking-harness-evolution.

Yike Wang, Huaisheng Zhu, Zhengyu Hu +7

Allen Institute for AI · Independent · University of Washington

Large Language Model Agents Test-Time Scaling

Ask PDF

Sep 1, 2026cs.SE

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16

Agent Harness

Ask PDF