Harnesses That Improve Themselves: A Unifying Framework for Self-Improving Agent Harnesses
Organizations: Google
Abstract
Whether a foundation model succeeds as an autonomous agent depends as much on its runtime software as on its weights. This runtime, the agent harness, manages context and memory, steers multi-turn control, executes tool actions in sandboxes, and verifies work before stopping. Because models differ in how they use tools and how they fail, a static harness tuned for one model or task often stalls, wastes tokens, or regresses on another. This bottleneck has driven a shift from prompt tuning to self-improving agent harnesses: systems that inspect their own execution traces, separate runtime bugs from model limits, and rewrite their own code. We present a unifying architectural and design framework for self-improving agent harnesses. The framework pairs four editable runtime components (context and state, agent control, action interfaces, and verification and recovery) with a four-stage loop that diagnoses failures, proposes code patches, validates candidates, and deploys updates. Using this framework, we show how optimizers trace errors across long trajectories and discover reusable runtime mechanisms such as pre-stop verifiers, loop breakers, and persistent memory. We further show why full-harness rewrites overfit or break across modules, and how modular updates with validation gates generalize across models while bounding token cost. To keep self-modification safe, we establish practical defenses against recurring failure modes including prompt bloat, benchmark gaming, and evaluator tampering. Finally, experiments on SWE-bench Lite validate this framework, showing that gated modular evolution outperforms prompt reflection and full-harness rewrites across models and repositories at a fraction of the token cost.
Figures & tables
| System (Domain) | Failure Attribution (Stage 1) | Optimizer & Editable Scope (Stage 2) | Validation & Deployment (Stages 3 and 4) | Emergent Mechanism & Takeaway |
| Part A: Trajectory-Grounded Hook and Module Optimizers | ||||
| Meta-Harness [ 19 ] (Coding, Math, Text) | Searches raw execution trajectories, scores, and prior code diffs | External coder: Rewrites full harness code, compactors, and verifiers | Selects top dev candidate; tests held-out task and model transfer | Raw trajectory access outperforms scalar scores; core structure transfers across models |
| Self-Harness [ 8 ] (Terminal, SWE, App APIs) | Clusters failed rollouts by compiler and verifier error signatures | Same-model: Patches pre-turn, post-tool, and pre-stop lifecycle hooks | Requires training gains with zero regression on validation split | Models invent tailored loop breakers, patch subagents, and PATH persistence |
| AutoSaddler [ 21 ] (Terminal, SWE) | Traces multi-turn rollouts to separate tool gaps from bad control | External coder: Edits tool wrappers, prompts, and retry control logic | Screens on failure batch, then branches from best validation ancestor | Multi-turn trace debugging outperforms shallow single-run reflection |
| HarnessCompass [ 22 ] (SWE Coding) | Pairs external trace logs with task-agent bottleneck self-reports | External coder: Edits scoped tool-interface and instruction harness modules | Enforces global generality checks before periodic module merges | Self-reports expose clunky tool interfaces missed by outcome logs |
| Part B: Modular, Archive, and Recursive Self-Improvers | ||||
| Failure Mode & Component | Root Cause | Observed Runtime Behavior | Warning Signs in Logs | Practical Safeguard |
| Part A: Inner-Harness Degradation (Context, Tools, Control, and Verification) | ||||
| 1. Unnecessary Rules & Prompt Bloat (Context & State) | Single-run reflection mistakes random slips for missing rules and appends without cleanup | Adds rules for violations that never occurred [ 50 ] ; instruction files grow endlessly [ 51 ] | Growing prompt length; zero validation gain; dropped user constraints [ 59 ] | Require repeated failures [ 8 ] ; use short “do-not” rules [ 55 ] ; prune unhelpful rules [ 20 ] |
| 2. Tool Overload & Confusion (Action Interface) | Overlapping tool or plugin schemas confuse action and argument selection | Large MCP lists and helper libraries cause wrong tool picks [ 36 , 75 ] and underperform a minimal shell [ 35 ] | Frequent schema errors; inconsistent tool choices; high initial prompt overhead | Use minimal shell and code interfaces [ 35 , 34 ] ; load tool schemas on demand [ 36 , 61 ] |
| 3. Conflicting Module Edits (Agent Control & Verification) | Editing loop limits, output filters, and stop checks at once breaks shared assumptions | Full-harness edits underperform the seed harness on held-out tasks [ 25 ] and bloat token usage [ 20 ] | Patches pass alone but fail together; spikes in turn count or early exits | Evolve modules separately with conflict-resolving merges [ 25 , 22 ] ; verify new code runs [ 24 ] |
| Part B: Sandbox, Evaluator, and Memory Exploits (Environment, Grading, and Persistence) | ||||
| 4. Benchmark Cheating & Overfitting (Sandbox & Grading) | Search rewards patches that leak git answers or memorize visible tests | Extracts fixes from git history [ 76 ] , ignores prompt bans [ 77 ] , or hard-codes visible test outputs [ 78 ] | Commands reading git history; sharp gap between visible and hidden pass rates | Strip git history and network access [ 76 ] ; grade on held-out test splits [ 78 , 20 ] |
| Principle / Standard | Practical Requirement & Why It Matters |
|---|---|
| Part A: Practical Safety Boundaries | |
| 1. Isolating the Evaluator & Logs | Keep grading scripts, test suites, and append-only execution logs in read-only storage outside the agent sandbox so the agent cannot inspect hidden tests or alter its own logs [ 47 , 48 ] . |
| 2. No Privilege Escalation | Enforce strict permission limits across self-edits so a modified harness can never grant itself broader file, network, or subprocess access than its parent had [ 83 , 57 ] . |
| 3. Securing Persistent Memory | Treat saved memories, learned rules, and generated plugins as untrusted inputs with tracked origins and per-skill and session token budgets; re-test archived harnesses when safety policies change [ 80 , 79 , 82 , 81 ] . |
| 4. Testing Harness Code | Pair evolved runtime modules with automated unit and branch-coverage tests that check LLM error handling and recovery paths [ 69 ] . |
| Part B: Evaluation Reporting Standards | |
| Harness Method & Search Scope | Held-Out Resolution Rate (% Solved) | Held-Out Cost & Exit Quality | |||||||
| Harness Method | Editable Scope | Search Rollouts | Turn Cap | In-Domain ( sympy/sympy ) | Cross-Repo OOD ( django/django ) | Combined ( ) | Avg. Tok. per Task | Tokens per Solved | Unverified OOD Exit % |
| Part A: Evolution on Frozen Gemini 3.8 Flash (Optimizer: Gemini 4 Carbon/Argon; Stage-3 runtime token cap ) | |||||||||
| Frozen Seed (Pass@1, ) | None (142-line seed) | 0 | 12 | 10.0 (2/20) | 0.0 (0/20) | 5.0 (2/40) | 114.8k | 2,296.7k | 15.0 |
| Frozen Seed (Best-of-3) | None ( test rank) | 0 | 12 | 5.0 (1/20) | 5.0 (1/20) | 5.0 (2/40) | 237.6k | 4,752.2k | 5.0 |
| Prompt-Only Reflection | RULES.md text only | 24 | 12 | 5.0 (1/20) | 0.0 (0/20) | 2.5 (1/40) | 60.5k | 2,420.2k | 30.0 |
| Unconstrained Full-Harness Rewrite | All 4 modules, ungated | 24 | 28 | 35.0 (7/20) | 50.0 (10/20) | 42.5 (17/40) | 451.6k | 1,062.6k | 5.0 |