The harness surrounding a language model is a central determinant of agent performance. Recent methods optimize harnesses by searching over complete programs, where individual mechanisms are difficult to isolate and reuse. We introduce PluginRSI, which represents a harness as a composition of atomized plugins and organizes harness evolution around these plugins. Individual plugins are improved independently and accumulated in a shared library, then recombined into new harnesses at each iteration. PluginRSI improves over existing harness optimization methods across software engineering, command-line interaction, and question-answering tasks. The resulting harnesses retain their advantage when transferred to other solver models without further optimization. The evolved plugin library accelerates subsequent optimization from the initial harness, which helps faster and higher convergence on unseen tasks. These results show that accumulating reusable mechanisms provides an effective basis for continued harness improvement.
Figures & tables
Figure 1: Comparison of harness optimization paradigms. (a) Prior approaches expose the complete harness Ht as the object of modification. (b) PluginRSI explores recursive optimization of plugin-parameterized harnesses, where each plugin encapsulates an independent mechanism that can be accumulated and reused.
Figure 2: Overview of PluginRSI. (1) Plugin mutation explores individual plugin changes in parallel while keeping the remaining harness fixed. (2) Harness recomposition selects plugins from the updated library and revises the workflow to construct a new harness candidate.
Held-In (400)
Held-Out (100)
Method
GPT-5.6 Terra
Kimi-K3
GLM-5.2
GPT-5.6 Terra
Kimi-K3
GLM-5.2
Expert-Curated Harnesses
ReAct (Zero-Shot)
38.25
41.00
37.75
39.00
41.00
41.00
ReAct (Few-Shot)
52.50
44.25
38.25
55.00
47.00
41.00
ACE
46.75
41.00
30.25
43.00
37.00
31.00
Automatically Optimized Harnesses w/ GPT-5.6 Terra
Table 1: Comparison on SWE-bench Verified ( Chowdhury et al., 2024 ) . Harnesses are optimized and selected on 400 held-in tasks, then fixed for evaluation across solver backbones. Values are resolve rates (%). † marks the score used for harness selection. Bold marks the best result in each column within each optimization setting.
Method
Held-In (59)
Held-Out (30)
ReAct (Zero-Shot)
47.5
50.0
ACE
54.2
56.7
GEPA
57.6
63.3
DGM
57.6
60.0
Meta-Harness
64.4
66.7
PluginRSI
69.5
73.3
Table 2: Results on Terminal-Bench-2.1 with Kimi-K3. Values are resolve rates (%). Bold marks the best result in each column.
Held-In (150)
Held-Out (250)
Method
Math
SciCite
Eco
Avg.
Fin
FS
Physics
Law
Med
Avg.
Overall
ReAct
0.480
0.540
0.420
0.480
0.280
0.620
0.140
0.320
0.300
0.332
0.388
GEPA
0.680
0.640
0.580
0.633
0.380
0.660
0.200
0.620
0.500
0.472
0.533
Meta-Harness
0.780
0.680
0.800
0.753
0.480
0.740
0.280
0.800
0.540
0.568
0.638
PluginRSI
0.880
0.780
0.800
0.820
0.560
0.720
0.280
0.820
0.560
0.588
0.675
Table 3: Results on QA benchmarks. We optimize harnesses on 150 tasks from three held-in domains. Both optimization and evaluation are on Kimi-K3. Values are accuracies rounded to 3 decimals. Bold indicates the best results.
Figure 3: Optimization dynamics. Held-in and held-out resolve rates of the best held-in harness against cumulative solver rollouts. PluginRSI yields higher performing harness with fewer solver rollouts, and held-in performances aligns better with held-out scores.
Held-In (400)
Held-Out (100)
Method
GPT-5.6 Terra
Kimi-K3 †
GLM-5.2
GPT-5.6 Terra
Kimi-K3
GLM-5.2
Meta-Harness
58.25
64.25
59.75
60.00
63.00
59.00
+ Plugin Library
58.25
65.50
55.25
58.00
64.00
56.00
+ Plugin Library w/o Initial Plugins
61.00
64.75
56.25
62.00
62.00
58.00
PluginRSI
58.00
66.75
59.50
64.00
69.00
64.00
w/o Initial Plugins
62.25
67.00
57.75
62.00
67.00
63.00
Table 4: Ablation study on SWE-bench Verified. Harnesses are optimized and selected with Kimi-K3 on 400 held-in tasks, then fixed for evaluation across solver backbones. Values are resolve rates (%) and † marks the score used for harness selection. Bold marks the best result in each column.
Figure 4: Plugin reuse effectiveness. On held-out tasks, optimizing from initial harness w/ the evolved plugin library accelerates convergence.
Figure 5: Plugin accumulation statistics. (a) Number of plugins in the plugin library. (b) Candidate implementation size during optimization. PluginRSI explores more complex harness structure by reusing discovered plugins, while the workflow implementation remains relatively brief.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Proposer Agent
Optimization iterations ( T )
15
Mutation candidates per iteration ( N )
8
Mutation minibatch size ( b )
20
Recomposition candidates per iteration
1
Solved:Unsolved tasks per minibatch
10:10
Appendix
Table 5: Hyperparameters.
Role
Model
Reasoning
Max Output Tokens
Proposer
GPT-5.6 Terra
xhigh
16,384
Proposer
Kimi-K3
xhigh
16,384
Solver
GPT-5.6 Terra
none
8,192
Solver
Kimi-K3
none
8,192
Solver
GLM-5.2
disabled
8,192
Appendix
Table 6: Model configurations for the proposer and solvers.
Method
Held-In (400)
Held-Out (100)
GEPA
61.00
62.00
DGM
58.00
55.00
SkillOPT ( Yang et al., 2026b )
54.75
60.00
A-Evolve ( Lin et al., 2026b )
61.75
57.00
AHE ( Lin et al., 2026a )
60.25
63.00
Meta-Harness
64.25
63.00
Appendix
Table 7: Results on SWE-bench Verified with Kimi-K3 as both proposer and solver. Values are resolve rates (%). Bold marks the best result in each column.
Source category
Papers and resources
Research papers
SWE-agent ( Yang et al., 2024 ) , Agentless ( Xia et al., 2024 ) , SWE-smith ( Yang et al., 2026a ) , and SWE-Bench ( Jimenez et al., 2024 ) .
SWE-Bench experiment repository and SWE-smith trajectory corpus .
Appendix
Table 8: Sources used to construct the seed plugin library. Repository and specification names link to their corresponding resources.
Category
Count
Functional coverage
Roles
20
Baseline coding and boundary enforcement; task analysis and context preparation; fault localization and diagnosis; patch planning, generation, and ranking; verification and finalization; experience extraction and plugin or workflow editing.
Tools
16
Terminal and shell execution; file access and repository navigation; text and symbol search; patch application; Git inspection; test discovery and execution; linting and failure inspection; patch submission.
Skills
31
Task analysis and repository orientation; skill routing and instruction loading; code localization and call-chain tracing; reproduction and debugging; patching and contract preservation; testing and completion checks; blocker diagnosis and recovery; lesson extraction and skill improvement; harness-authoring guidance.
Memory
8
A baseline experience bank and seven structured record types: trajectories, task outcomes, failure signatures, lesson candidates, counterexamples, component credit, and skill evaluations.
Total
75
Appendix
Table 9: Composition of the seed plugin library. Coverage is summarized by function.
Figure 6: Python implementation of the initial harness workflow.
Figure 7: Sub-domain distribution of held-in and held-out tasks on Swe-Bench-Verified.
Figure 8: Examples of plugin and harness manifests. (a) plugin.yaml defines a terminal tool and its instance configuration. (b) harness.yaml assembles a harness by mapping workflow aliases to versioned plugins and their configurations.
Category
Interface
Function
Role
render(context)
Returns a prompt string constructed from the supplied context.
Tool
schema() execute(arguments)
Exposes the tool-call schema and asynchronously executes an operation, returning a ToolResult .
Skill
load(context)
Returns a SkillContent containing instructions and associated resources.
Memory
update(experience) retrieve(query)
Asynchronously stores an experience and retrieves a list of MemoryItem objects.
Appendix
Table 10: Plugin interfaces exposed to the workflow.
Category
Discovered mechanism
Before
After
Role
Repair-scope control
10/20
13/20
Discriminative checks and lifecycle continuity
10/20
13/20
Skill
Semantic source preservation
10/20
13/20
Explicit precedence and representation checks
10/20
13/20
Public-path checks with reachability controls
10/20
14/20
Memory
Incremental evidence retrieval
10/19
14/19
Appendix
Table 11: Selected plugin mutations and their screening results. Values are resolved tasks before and after mutation on each candidate’s minibatch, using cached outcomes for the original harness. For incremental evidence retrieval, one verifier timeout leaves 19 tasks with valid outcomes in both evaluations.
Figure 9: Repair-scope control in the coder role. The mutated instructions specify where to place guards, transactions, or cleanup when repairing persistent mutations.
Figure 10: Semantic source preservation in the debugging skill. The mutated instructions require an explicit decision about the correct behavior when the intended source is absent.
Figure 11: Incremental evidence retrieval in the memory plugin. Mutation replaces retrieval by lexical overlap with deduplication and summaries of entries not previously returned. Code excerpts are reformatted and omit supporting logic.
Figure 12: Failure-aware terminal feedback. The mutated tool interprets command output and tracks repeated failures to guide subsequent checks. Code excerpts are reformatted and omit supporting logic.
Figure 13: Token cost analysis. (a) Total token usage per solver rollout, including solver and proposer tokens. (b) Solver tokens per rollout. (c) Proposer tokens per proposal, with plugin mutation and harness recomposition reported separately.
An agent harness is the code that organizes context, maintains state, and coordinates tool calls for a language model. We study how to improve the harness under a limited evaluation budget while keeping model weights fixed. Our method, MESH-Harness, organizes each harness into functional modules with explicit role-specific interfaces, allowing alternative implementations of each module to be substituted and recombined. It uses shared module representations and full-covariance LinUCB to score candidate combinations based on predicted performance and exploration value. Mixed-start coordinate ascent selects complete configurations for evaluation without enumerating the combinatorial space. Validation traces then guide local code edits, and the resulting candidates are incorporated into fixed-capacity role-specific pools for subsequent recombination. On text tasks, retrieval-augmented mathematical reasoning, code generation, and interactive scientific tasks, MESH-Harness outperforms Meta-Harness by 5.70, 7.01, 2.00, and 5.00 points, respectively, under matched candidate-evaluation budgets. Iterative harness optimization improves MESH-Harness by 5.63-7.79 points over its first-round configurations. For the reported configurations, aggregate test-time cost is 44.2% lower than that of Meta-Harness, while total cost including search is 14.6% lower. These results show that combining module-level design reuse with feedback-driven compositional search can systematically improve agent harnesses while keeping overall optimization cost under control.
Zhiwei Shang, Yu Huo, Mingrong Gong +6
The Chinese University of Hong Kong, Shenzhen · DeepWisdom · The Chinese University of Hong Kong +3
Automating the search for effective harnesses is an important step toward enabling agents to recursively self-improve. Existing harness optimizations typically produce a single global harness that is applied uniformly across task instances. However, a harness that works well on average may not be optimal for every instance. We introduce Turbo Harness, a framework that can adapt a globally optimized harness to each instance by reusing information generated during the original optimization process. Specifically, Turbo Harness recycles artifacts produced during a completed global harness optimization run, and summarizes them into a structured playbook. We train a harness editor to leverage this prior optimization experience to generate instance-specific patches to the global harness. At inference time, the editor uses the instance and the playbook to construct a tailored harness in which the execution model operates. Through numerical experiments, we show that Turbo Harness consistently outperforms existing harness optimization baselines across seven benchmarks spanning interactive agent tasks, software engineering, and long-horizon terminal tasks.
Tunyu Zhang, Hao Wang, Kai Xu +1
Rutgers University · Red Hat AI Innovation · MIT-IBM Watson AI Lab
AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.