Automated harness optimization improves agents by learning from trajectories and task feedback. Existing offline methods use reference answers to optimize harnesses for reuse on held-out tasks. However, many real-world tasks are open-ended, while long runs expose new difficulties that offline-optimized harnesses may not address. To study harness optimization in these settings, we introduce PerfRace, a benchmark for long-running performance optimization of software projects, and investigate whether agents can use task-specific experience to recursively improve their harnesses during an ongoing task. To this end, we introduce BOXBOX, which enables agent-triggered harness revision and kernel-managed hot reloading while preserving execution context. We evaluate BOXBOX on PerfRace under matched test-time budgets, comparing it with Frozen Harness, offline Harness Evolution, and Harness Scaling. BOXBOX achieves higher speedup with more efficient use of task time and model cost, reaching a Speedup@4h of 1.70x, compared with 1.54x for the strongest baseline.
Figures & tables
Figure 1: Performance of BoxBox and baselines on 50 PerfRace test tasks, with three runs per method and task. Curves show aggregate best-so-far correct speedup against task time 1(a) and mean cumulative model cost per run 1(b) , sampled at matching task times, with shaded bands indicating 95% fixed-task run-resampling intervals. BoxBox achieves higher speedup with greater time and cost efficiency than the baselines.
Figure 2: Overview of PerfRace . (a) Construction discovers projects from GitHub coding-agent activity, applies a two-stage Filter comprising a deterministic precheck and source-based assessment, and uses Agent–Human Pair Verification to produce a PerfRace task. (b) Task format separates the agent workspace from hidden evaluation files. (c) The evaluation loop repeatedly submits workspace snapshots to the Judge and tracks best-so-far correct speedup.
Figure 3: Composition of PerfRace : seven workload domains (inner ring), 100 tasks (outer ring), and representative project–workload pairs (right).
Figure 4: BoxBox mechanism. The agent requests a review on demand, triggering a trap that transfers control to the kernel. The kernel then launches the review, validates revisions, and hot-reloads the runtime with preserved execution context and rollback if activation fails.
Method
Speedup@4h
Mean model cost
Speedup/\uparrow$
Frozen
1.442
$23.22
0.2142
HE
1.281
$24.56
0.1243
HS
1.539
$12.98
0.2872
BoxBox
1.702
$16.43
0.4886
Table 1: Results on 50 test tasks with three four-hour runs per task, reporting Speedup@4h , mean model cost per run, and mean task-level speedup per dollar.
Method
Speedup@4h
Mean model cost
Speedup/$
w/o Agent Triggering
1.508
$21.58
0.1338
w/o Full Editing
1.349
$16.07
0.1132
Full BoxBox
1.584
$15.07
0.3452
Table 2: Ablation study of BoxBox . We examine agent-controlled triggering and full-harness editing in terms of task performance ( Speedup@4h ), model cost, and cost efficiency (Speedup/$).
Figure 5: Harness optimization analysis. 5(a) Number of reviews per run across 150 test runs, including agent-triggered and fault-recovery reviews. 5(b) Per-run review time as a share of the four-hour task budget and review cost as a share of total model cost within that budget; dots represent runs, and boxes show the median and interquartile range. 5(c) Modification types across a collection of 620 applied, nonempty harness updates from BoxBox runs.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Uncached input
Cached input
Output
GPT-5.6-Luna
$1.00
$0.10
$6.00
GPT-5.6-Terra
$2.50
$0.25
$15.00
Appendix
Table 3: Model token rates used for cost calculation (USD per million tokens).
Stage
Tasks
Evaluated runs
Budget per run
Cost ($)
Training
40
24 iterations × 5 tasks
45 min
200.34
Validation
10
3 harnesses × 3 runs/task
2 h
1,176.12
Test
50
h24, 3 runs/task
4 h
3,683.34
Appendix
Table 4: HE configuration and recorded model costs. Training and validation costs include retries and reruns and are lower bounds where usage records are missing.
Figure 6: HE training and validation performance. 6(a) Mini-batch training speedup, with aggregate scores for each epoch. 6(b) Validation speedup of harness checkpoints h8 , h16 , and h24 . Speedups are aggregated as in Equation 2 .
Figure 8: HS optimization trajectory on kitops--minilm-pack . Annotations show selected harness revisions and an agent session containing 113 consecutive checks that produced no new performance measurements.
Figure 9: BoxBox optimization trajectory on ccc--compile-speed . Points show correct evaluation results and the line shows best-so-far speedup. Annotations mark harness review outcomes and task-code changes; numerical gains are absolute and relative increases over the preceding best-so-far speedup.
Figure 10: BoxBox optimization trajectory on fluidsynth--bagatelles , using the plotting conventions of Figure 9 . Reviews extend command waiting and add batch benchmarking.
Case
Category
Task ID
Modification
A.4.2
Review feedback
n8n-mcp--tool-batch-speed
Redirect the next experiment toward SQLJS conversion and statement overhead.
A.4.2
Prompt rules and skills
sum--text-attest
Require alternating A/B measurements when apparent gains are close to noise.
A.4.2
Tool functionality
rust-clippy--cargo-package-lint
Extend internal command polling to reduce model-directed waiting turns.
Add a message-count-based context-compaction path.
A.4.2
Fault fix
rust-clippy--cargo-package-lint
Replace an exception on shell exit with a session-recovery call.
Appendix
Table 5: Representative harness updates grouped by modification type. Case numbers refer to the detailed analyses below; task IDs follow the catalog in Table 7 .
Live GitHub discovery followed by agent–human Pair Verification
Iterative correctness and performance feedback from an external Judge
AutoLab ( Xu et al., 2026 )
Diverse metric-driven optimization tasks
Expert-curated optimization tasks
Local evaluation during the task; held-out final evaluation
EdgeBench ( Zhu et al., 2026 )
Cross-domain iterative problem-solving tasks
Expert-curated day-scale tasks
Local inner loop; repeated submission-gated Judge feedback
FrontierSWE ( Chu et al., 2026 )
Ultra-long-horizon software engineering and research tasks
Real-world problems curated with academic and industry experts
Final graded submission after up to 20 hours
Terminal-Bench ( Merrill et al., 2026 )
General terminal tasks
Crowdsourced tasks with multi-stage human review
Final-only test-based evaluation
GSO ( Shetty et al., 2025 )
Microbenchmark-based repository optimization
Repository commit histories and generated performance tests
Final-only microbenchmark-based evaluation
Appendix
Table 6: Comparison with related benchmarks. Microbenchmarks measure targeted operations (for example, selected API calls), whereas end-to-end workloads measure complete project-use workflows.
Figure 16: Distribution of primary programming languages, packaged source sizes, and packaged source file counts across PerfRace projects.
Task
Project and purpose
License
Fixed workload
aiidateam-aiida-core--battery-archive-import
aiidateam/aiida-core Scientific workflow and provenance platform
MIT
AiiDA import of a public battery-cycling provenance archive
anyrender--openmoji
DioxusLabs/anyrender Rust 2D drawing abstraction
Apache-2.0
AnyRender batch rasterization of the OpenMoji SVG corpus
apalache--zcash-sync
apalache-mc/apalache Symbolic model checker for TLA+ and Quint
Apache-2.0
Apalache’s bounded safety check for the Zcash v2 block-download scheduler
arrow-rs--variant-wikidata
apache/arrow-rs Rust implementation of Apache Arrow
Apache-2.0
apache/arrow-rs while encoding Wikidata entities as a Variant column
bevy-voxel-world--terrain-streaming
splashdust/bevy_voxel_world Voxel-world plugin for Bevy
Apache-2.0
splashdust/bevy_voxel_world terrain generation and chunk meshing
bm25s--trec-covid-index
xhluca/bm25s BM25 search library
MIT
bm25s indexing and persistence for the TREC-COVID research collection
Appendix
Table 7: Complete catalog of the 100 PerfRace tasks.
Signal
Repositories
Branch push by a coding-agent account
112,127
Pull request opened by a coding-agent account
6,390
Appendix
Table 8: GitHub activity signals used for Discovery.
Result
Projects
Fewer than 10 GitHub stars
93,992
Accessibility
1,134
Setup path
212
Runtime fit
359
Optimization potential
4,462
Evaluable process
3,066
Appendix
Table 9: First failed check during Filter and the final selection result.
Figure 17: Distribution of curation time, agent cost, and agent–human interactions across the 100 accepted tasks.
Figure 18: Pair Verification alternates between agent-led task construction and human review at the Realism, Reliability, and Integrity gates.
Figure 19: Candidates rejected at each Pair Verification stage, with candidates lacking a recorded transition into Build & Calibrate assigned to Explore & Design.
Input source
Examples
Tasks
Released source code, model, or asset
Linux source tree, OpenMoji assets, released ML models
Table 10: Sources of workload inputs across the 100 accepted tasks.
CV range
Tasks
0%≤CV<2%
20
2%≤CV<6%
64
6%≤CV<7%
8
7%≤CV<8%
3
8%≤CV<9%
4
9%≤CV<10%
0
Appendix
Table 11: Reliability and integrity evidence: final CVs for the 99 wall-clock tasks and red-team tests for all 100 tasks, with one test assigned conservatively when an exact count cannot be recovered.
Figure 20: Distribution of complete baseline Judge evaluation time across the 100 final task packages, with a mean of 6.1 minutes, a median of 4.9 minutes, and a 20-minute evaluation limit.
Figure 21: Performance improvements from additional optimization time, based on three four-hour runs for each of 88 PerfRace tasks. Left: increase in the overall score at four hours relative to two and three hours; horizontal lines show 95% bootstrap intervals. Right: number of tasks in each improvement range.
Outcome
Continuation
No harness change
Resume the current runtime with the preserved task context.
Successful update
Continue under the revised harness with transferred context, the existing terminal session, and the same project workspace.
Update fails
Retain or restore the previous runtime and its preserved context.
Runtime fault
Enter recovery and review using the retained execution records; restart the terminal while preserving project files.
Appendix
Table 12: Task continuation across review and runtime-update outcomes.
Automating the search for effective harnesses is an important step toward enabling agents to recursively self-improve. Existing harness optimizations typically produce a single global harness that is applied uniformly across task instances. However, a harness that works well on average may not be optimal for every instance. We introduce Turbo Harness, a framework that can adapt a globally optimized harness to each instance by reusing information generated during the original optimization process. Specifically, Turbo Harness recycles artifacts produced during a completed global harness optimization run, and summarizes them into a structured playbook. We train a harness editor to leverage this prior optimization experience to generate instance-specific patches to the global harness. At inference time, the editor uses the instance and the playbook to construct a tailored harness in which the execution model operates. Through numerical experiments, we show that Turbo Harness consistently outperforms existing harness optimization baselines across seven benchmarks spanning interactive agent tasks, software engineering, and long-horizon terminal tasks.
Tunyu Zhang, Hao Wang, Kai Xu +1
Rutgers University · Red Hat AI Innovation · MIT-IBM Watson AI Lab
Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro
Zhuoshi Pan, ZhiYuan Feng, Yukuo Cen +6
Tsinghua University · Z.AI Co., Ltd. · Shanghai Jiao Tong University +2
AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.