cs.AISep 26, 2026
SavePluginRSI: Recursive Improvement of Agent Harnesses with Reusable Plugins
Organizations: University of Science and Technology of China · Meituan · Wuhan University · National University of Singapore
Abstract
The harness surrounding a language model is a central determinant of agent performance. Recent methods optimize harnesses by searching over complete programs, where individual mechanisms are difficult to isolate and reuse. We introduce PluginRSI, which represents a harness as a composition of atomized plugins and organizes harness evolution around these plugins. Individual plugins are improved independently and accumulated in a shared library, then recombined into new harnesses at each iteration. PluginRSI improves over existing harness optimization methods across software engineering, command-line interaction, and question-answering tasks. The resulting harnesses retain their advantage when transferred to other solver models without further optimization. The evolved plugin library accelerates subsequent optimization from the initial harness, which helps faster and higher convergence on unseen tasks. These results show that accumulating reusable mechanisms provides an effective basis for continued harness improvement.
Figures & tables
Figure 1: Comparison of harness optimization paradigms. (a) Prior approaches expose the complete harness as the object of modification. (b) PluginRSI explores recursive optimization of plugin-parameterized harnesses, where each plugin encapsulates an independent mechanism that can be accumulated and reused.
Figure 2: Overview of PluginRSI. (1) Plugin mutation explores individual plugin changes in parallel while keeping the remaining harness fixed. (2) Harness recomposition selects plugins from the updated library and revises the workflow to construct a new harness candidate.
| Held-In (400) | Held-Out (100) | |||||
| Method | GPT-5.6 Terra | Kimi-K3 | GLM-5.2 | GPT-5.6 Terra | Kimi-K3 | GLM-5.2 |
| Expert-Curated Harnesses | ||||||
| ReAct (Zero-Shot) | 38.25 | 41.00 | 37.75 | 39.00 | 41.00 | 41.00 |
| ReAct (Few-Shot) | 52.50 | 44.25 | 38.25 | 55.00 | 47.00 | 41.00 |
| ACE | 46.75 | 41.00 | 30.25 | 43.00 | 37.00 | 31.00 |
| Automatically Optimized Harnesses w/ GPT-5.6 Terra | ||||||
Table 1: Comparison on SWE-bench Verified ( Chowdhury et al., 2024 ) . Harnesses are optimized and selected on 400 held-in tasks, then fixed for evaluation across solver backbones. Values are resolve rates (%). marks the score used for harness selection. Bold marks the best result in each column within each optimization setting.
| Method | Held-In (59) | Held-Out (30) |
|---|---|---|
| ReAct (Zero-Shot) | 47.5 | 50.0 |
| ACE | 54.2 | 56.7 |
| GEPA | 57.6 | 63.3 |
| DGM | 57.6 | 60.0 |
| Meta-Harness | 64.4 | 66.7 |
| PluginRSI | 69.5 | 73.3 |
Table 2: Results on Terminal-Bench-2.1 with Kimi-K3. Values are resolve rates (%). Bold marks the best result in each column.
| Held-In (150) | Held-Out (250) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Math | SciCite | Eco | Avg. | Fin | FS | Physics | Law | Med | Avg. | Overall |
| ReAct | 0.480 | 0.540 | 0.420 | 0.480 | 0.280 | 0.620 | 0.140 | 0.320 | 0.300 | 0.332 | 0.388 |
| GEPA | 0.680 | 0.640 | 0.580 | 0.633 | 0.380 | 0.660 | 0.200 | 0.620 | 0.500 | 0.472 | 0.533 |
| Meta-Harness | 0.780 | 0.680 | 0.800 | 0.753 | 0.480 | 0.740 | 0.280 | 0.800 | 0.540 | 0.568 | 0.638 |
| PluginRSI | 0.880 | 0.780 | 0.800 | 0.820 | 0.560 | 0.720 | 0.280 | 0.820 | 0.560 | 0.588 | 0.675 |
Table 3: Results on QA benchmarks. We optimize harnesses on 150 tasks from three held-in domains. Both optimization and evaluation are on Kimi-K3. Values are accuracies rounded to 3 decimals. Bold indicates the best results.
Figure 3: Optimization dynamics. Held-in and held-out resolve rates of the best held-in harness against cumulative solver rollouts. PluginRSI yields higher performing harness with fewer solver rollouts, and held-in performances aligns better with held-out scores.
| Held-In (400) | Held-Out (100) | |||||
| Method | GPT-5.6 Terra | Kimi-K3 † | GLM-5.2 | GPT-5.6 Terra | Kimi-K3 | GLM-5.2 |
| Meta-Harness | 58.25 | 64.25 | 59.75 | 60.00 | 63.00 | 59.00 |
| + Plugin Library | 58.25 | 65.50 | 55.25 | 58.00 | 64.00 | 56.00 |
| + Plugin Library w/o Initial Plugins | 61.00 | 64.75 | 56.25 | 62.00 | 62.00 | 58.00 |
| PluginRSI | 58.00 | 66.75 | 59.50 | 64.00 | 69.00 | 64.00 |
| w/o Initial Plugins | 62.25 | 67.00 | 57.75 | 62.00 | 67.00 | 63.00 |
Table 4: Ablation study on SWE-bench Verified. Harnesses are optimized and selected with Kimi-K3 on 400 held-in tasks, then fixed for evaluation across solver backbones. Values are resolve rates (%) and marks the score used for harness selection. Bold marks the best result in each column.
Figure 4: Plugin reuse effectiveness. On held-out tasks, optimizing from initial harness w/ the evolved plugin library accelerates convergence.
Figure 5: Plugin accumulation statistics. (a) Number of plugins in the plugin library. (b) Candidate implementation size during optimization. PluginRSI explores more complex harness structure by reusing discovered plugins, while the workflow implementation remains relatively brief.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
| Proposer Agent | |
| Optimization iterations ( ) | 15 |
| Mutation candidates per iteration ( ) | 8 |
| Mutation minibatch size ( ) | 20 |
| Recomposition candidates per iteration | 1 |
| Solved:Unsolved tasks per minibatch | 10:10 |
Table 5: Hyperparameters.
| Role | Model | Reasoning | Max Output Tokens |
|---|---|---|---|
| Proposer | GPT-5.6 Terra | xhigh | 16,384 |
| Proposer | Kimi-K3 | xhigh | 16,384 |
| Solver | GPT-5.6 Terra | none | 8,192 |
| Solver | Kimi-K3 | none | 8,192 |
| Solver | GLM-5.2 | disabled | 8,192 |
Table 6: Model configurations for the proposer and solvers.
| Method | Held-In (400) | Held-Out (100) |
|---|---|---|
| GEPA | 61.00 | 62.00 |
| DGM | 58.00 | 55.00 |
| SkillOPT ( Yang et al., 2026b ) | 54.75 | 60.00 |
| A-Evolve ( Lin et al., 2026b ) | 61.75 | 57.00 |
| AHE ( Lin et al., 2026a ) | 60.25 | 63.00 |
| Meta-Harness | 64.25 | 63.00 |
Table 7: Results on SWE-bench Verified with Kimi-K3 as both proposer and solver. Values are resolve rates (%). Bold marks the best result in each column.
| Source category | Papers and resources |
|---|---|
| Research papers | SWE-agent ( Yang et al., 2024 ) , Agentless ( Xia et al., 2024 ) , SWE-smith ( Yang et al., 2026a ) , and SWE-Bench ( Jimenez et al., 2024 ) . |
| Agent implementations | mini-SWE-agent , Aider , DeepSeek Harness , and OpenHands Software Agent SDK . |
| Skill instructions and organization | Agent Skills specification , Anthropic Skills , Superpowers , Addy Osmani Agent Skills , and OpenHands Extensions . |
| Trajectories and evaluation records | SWE-Bench experiment repository and SWE-smith trajectory corpus . |
Table 8: Sources used to construct the seed plugin library. Repository and specification names link to their corresponding resources.
| Category | Count | Functional coverage |
|---|---|---|
| Roles | 20 | Baseline coding and boundary enforcement; task analysis and context preparation; fault localization and diagnosis; patch planning, generation, and ranking; verification and finalization; experience extraction and plugin or workflow editing. |
| Tools | 16 | Terminal and shell execution; file access and repository navigation; text and symbol search; patch application; Git inspection; test discovery and execution; linting and failure inspection; patch submission. |
| Skills | 31 | Task analysis and repository orientation; skill routing and instruction loading; code localization and call-chain tracing; reproduction and debugging; patching and contract preservation; testing and completion checks; blocker diagnosis and recovery; lesson extraction and skill improvement; harness-authoring guidance. |
| Memory | 8 | A baseline experience bank and seven structured record types: trajectories, task outcomes, failure signatures, lesson candidates, counterexamples, component credit, and skill evaluations. |
| Total | 75 |
Table 9: Composition of the seed plugin library. Coverage is summarized by function.
Figure 6: Python implementation of the initial harness workflow.
Figure 7: Sub-domain distribution of held-in and held-out tasks on Swe-Bench-Verified.
Figure 8: Examples of plugin and harness manifests. (a) plugin.yaml defines a terminal tool and its instance configuration. (b) harness.yaml assembles a harness by mapping workflow aliases to versioned plugins and their configurations.
| Category | Interface | Function |
|---|---|---|
| Role | render(context) | Returns a prompt string constructed from the supplied context. |
| Tool | schema() execute(arguments) | Exposes the tool-call schema and asynchronously executes an operation, returning a ToolResult . |
| Skill | load(context) | Returns a SkillContent containing instructions and associated resources. |
| Memory | update(experience) retrieve(query) | Asynchronously stores an experience and retrieves a list of MemoryItem objects. |
Table 10: Plugin interfaces exposed to the workflow.
| Category | Discovered mechanism | Before | After |
|---|---|---|---|
| Role | Repair-scope control | 10/20 | 13/20 |
| Discriminative checks and lifecycle continuity | 10/20 | 13/20 | |
| Skill | Semantic source preservation | 10/20 | 13/20 |
| Explicit precedence and representation checks | 10/20 | 13/20 | |
| Public-path checks with reachability controls | 10/20 | 14/20 | |
| Memory | Incremental evidence retrieval | 10/19 | 14/19 |
Table 11: Selected plugin mutations and their screening results. Values are resolved tasks before and after mutation on each candidate’s minibatch, using cached outcomes for the original harness. For incremental evidence retrieval, one verifier timeout leaves 19 tasks with valid outcomes in both evaluations.
Figure 9: Repair-scope control in the coder role. The mutated instructions specify where to place guards, transactions, or cleanup when repairing persistent mutations.
Figure 10: Semantic source preservation in the debugging skill. The mutated instructions require an explicit decision about the correct behavior when the intended source is absent.
Figure 11: Incremental evidence retrieval in the memory plugin. Mutation replaces retrieval by lexical overlap with deduplication and summaries of entries not previously returned. Code excerpts are reformatted and omit supporting logic.
Figure 12: Failure-aware terminal feedback. The mutated tool interprets command output and tracks repeated failures to guide subsequent checks. Code excerpts are reformatted and omit supporting logic.
Figure 13: Token cost analysis. (a) Total token usage per solver rollout, including solver and proposer tokens. (b) Solver tokens per rollout. (c) Proposer tokens per proposal, with plugin mutation and harness recomposition reported separately.
More in this topic
Agent Harness Optimization
Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.
See all 183 papers