Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?

Kidus Seyoum; Ajay Mittur

cs.SESep 24, 2026

Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?

Authors: Kidus Seyoum, Ajay Mittur

Organizations: NVIDIA

Abstract

Agent behavior depends on the harness surrounding a language model, but it remains unclear whether language models can reliably improve such harnesses for hardware-design tasks. We study automatic harness evolution around a fixed subject model on 12 proprietary design-verification root-cause localization tasks. Across five trials per task, automatically evolved harnesses increased completed attempts by 71-76% and any-hit task coverage by 80-100%, while total correct attempts improved by only 18-24%. The strongest success reproducible at least twice result improved by one task, and later candidates exchanged gains across tasks rather than preserving them. An auxiliary candidate improved on a four-task validation set excluded from search but tied its baseline on a subsequent 12-task replay containing both search and validation tasks, so the selected gain did not persist across the full pool. Across the tested lineage, useful search, evidence, and finalization behaviors appeared in different candidates but did not consistently consolidate into a single harness that dominated across tasks and metrics. In a separate CVDP cross-benchmark case study, an automatically evolved defined-width repair harness produced 35.6% more functional passes than its 142-task reference baseline; the final functional verifier scored completed outputs but was not shown to the subject agent during repair. These results support archive-aware selection when evolution yields complementary specializations without consistent consolidation.

Figures & tables

Figure 1: A candidate harness and task instantiate an isolated attempt around a fixed subject model. A separate outer loop proposes branches, verifies activation, evaluates candidates, confirms promising results, and retains both the incumbent and archive.

Evaluation track	Task type	Scoring criterion	Role in the study
Proprietary design-verification benchmark	12 root-cause localization tasks	Format-normalized function exact match	Primary within-domain diagnostic evaluation
Selected CVDP subset	12 RTL generation and repair tasks	Executable functional verifier	Selected-subset phase of the cross-benchmark case study
142-task CVDP follow-up	142 RTL generation and repair tasks	Executable functional verifier	Baseline-to-evolved comparison in the cross-benchmark case study

Table 1: Evaluation tracks and their roles in the study.

Figure 2: The primary benchmark evaluates causal diagnosis over mixed verification artifacts, while CVDP evaluates completed RTL outputs with executable functional checks. The final functional verifier was not shown to the subject agent during repair. The tracks support different claims and are not numerically interchangeable.

Harness	First trial	Success $\geq$ 2/5	Pass@5	Correct attempts	Completed
Kimi K3 baseline	2/12	5/12	5/12	17/60	34/60
Automatically evolved evidence-grounded	6/12	6/12	9/12	20/60	58/60
Automatically evolved batched-causal	5/12	5/12	10/12	21/60	60/60

Table 2: Repeated evaluation results for the baseline and selected evolved harnesses.

Figure 3: Successful trials out of five for each task and harness. Aggregate gains conceal regressions and complementary task-level strengths.

Figure 4: MiniMax task success rate and mean graded task score across 72 completed generations; the search was then stopped after no further meaningful gain. A task counts as successful when $s_{t}\geq 0.8$ ; $\bar{s}$ averages the 12 graded task scores. The two final values both round to 0.417. Promotion labels identify the principal mechanism added at each accepted step.

Stage	Principal addition	Intended effect
Evidence-first	Bounded orientation, literal search, anchored reads, forced finalization	Reduce speculative search and timeouts
Coverage ledger	Observed-text and inspected-file ledger with search/read budgets	Balance premature fixation and uncontrolled exploration
Evidence-grounded	Queries restricted to observed text; bounded sequential reads	Prevent invented queries and expensive paging
Causal commit	Multiple evidence types; source-to-propagation-to-detection ordering	Avoid naming the checker as the cause
Batched causal	Independent calls per turn plus standardized read windows	Gather more evidence within the turn budget

Table 3: Audited mechanism lineage. Later stages inherit earlier controls.

Figure 5: Best branch-attempt score by task and iteration for two evolution methods. The methods and run lengths differ, so this is a descriptive view of archive activity rather than a matched comparison.

Figure 6: Candidate archives cover more tasks than individual harnesses. In the large branching experiment, the 4/12 bar is the best archived individual and was not promoted; the promoted incumbent remained at 3/12. Archive and union bars represent portfolio coverage, not the score of a composed generic harness.

Domain-context treatment	Mean candidate pass rate	Best result	Interpretation
Static curated context	16.7%	4/12	Matched standalone control
Live knowledge retrieval	17.5%	4/12	Matched standalone control
Low-density injection	10.4%	3/12	No reliable improvement
Specificity instruction	10.3%	3/12	No demonstrated improvement
Heavy domain context	10.6%	2/12	Content changed; score stayed flat

Table 4: Domain-context treatments and best observed benchmark results.

Figure 7: Functional-pass progression on the selected 12-task CVDP subset. The candidate harnesses were automatically evolved; later candidates use explicit routing and previously scored task artifacts, so the 10/12 endpoint is not a generic-harness result. Functional passes were measured after candidate outputs were complete.

CVDP harness condition	Subject LLM	Functional passes
Reference baseline: verification loop	GLM-5.2	45/142 (31.7%)
Automatically evolved: defined-width repair	GLM-5.2	61/142 (43.0%)

Table 5: Functional-verifier results for the reference baseline and automatically evolved repair harness on the 142-task CVDP follow-up.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Evidence acquisition, coverage tracking, causal analysis, and final commitment within one subject attempt.

Figure 9: Candidate branches pass through materialization, static validation, activation checks, evaluation, and experiment-specific retention.

Design-verification task subfamily	Tasks	Typical failure or reasoning pattern
Local testbench or resource accounting	T1, T2, T3, T4, T5, T7	Guards, counters, resource accounting, and tracing a local symptom to its producer
RTL or interface-state update	T8, T12	Assignment semantics, state transitions, and calling-logic behavior
Simulation, launch, or cross-layer behavior	T6, T9, T10, T11	Stimulus, checking, memory modes, restore behavior, and multi-layer failures

Appendix

Table 6: Provisional design-verification task subfamilies represented in the benchmark.

Treatment	Knowledge treatment	Mean candidate pass rate	Best result
Static curated DV context	Documentation in proposer context	16.7%	4/12
Live DV retrieval	Proposer could retrieve specialized knowledge	17.5%	4/12
Low-density injection	About 11 terms per 1,000 words	10.4%	3/12
Specificity instruction	About 18 terms per 1,000 words	10.3%	3/12
Heavy DV context	About 25 terms per 1,000 words	10.6%	2/12

Appendix

Table 7: Domain-context treatments and their best observed scores.

Configuration	Evaluated configurations	Mean pass rate	Best score	Input tokens	Agent time
Empty-policy baselines	4	14.6%	3/12	5.3–6.7M	about 1,250 s
Evolved domain scaffolds	58	n.r.	3/12	7.4–8.2M	about 1,600 s

Appendix

Table 8: Empty-policy baselines in the incremental domain-context experiment.

Model	Passed attempts / 36	Timeouts
GLM 5.2	12/36	6
Kimi K2.6	6/36	17
Nemotron Ultra	4/36	4
MiniMax	2/36	23

Appendix

Table 9: Cross-model descriptive baseline results.

Explore similar work

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses
Aug 3, 2026Luan Zhang, Ruochen Zhou, Dandan Song +9Evolving Harness Agent Harness
Beijing Institute of Technology, China · City University of Hong Kong, China · Independent, China
Rethinking the Evaluation of Harness Evolution for Agents
Jul 14, 2026Yike Wang, Huaisheng Zhu, Zhengyu Hu +7Large Language Model Agents Test-Time Scaling
Allen Institute for AI · Independent · University of Washington
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16Agent Harness

Aug 3, 2026, cs.LGJ/K move · Enter open · S save

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Luan Zhang, Ruochen Zhou, Dandan Song, Zhengyu Chen+8

Beijing Institute of Technology, China · City University of Hong Kong, China · Independent, China

Evolving Harness Agent Harness Evolutionary Optimization Methods

Open paper AskPDF arXiv

Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints on evolution, restricting modifications to task-agnostic harness changes that generalize beyond the evolution tasks. It then augments trajectory-derived evidence with proactive first-person feedback from the agent about harness usage, yielding richer signals for evolution. Finally, it decouples the optimization of different harness components before consolidating them into a unified harness, reducing cross-component interference while preserving component synergy. On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54% to 66% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency. In addition, the evolved harness transfers effectively to held-out tasks and other models, demonstrating substantially stronger generalization than prior automatic harness evolution methods.

Abstract

Explore similar work

HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

Rethinking the Evaluation of Harness Evolution for Agents

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?