cs.CLMar 26, 2026
SaveNatural-Language Agent Harnesses
Organizations: Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology (Shenzhen)
Abstract
Agent performance is strongly shaped by the surrounding harness: the external execution system around a model that organizes a task run. Yet this logic is usually buried in tightly coupled controller code, which makes harnesses hard to inspect, compare, transfer, and ablate. This paper asks whether the reusable design pattern of an agent harness can be represented as an executable natural-language object. We introduce Natural-Language Agent Harnesses (NLAHs), editable documents that describe run-level harness policy, and Intelligent Harness Runtime (IHR), a shared runtime that interprets these documents into agent calls, handoffs, state updates, validation gates, and artifact contracts. Across coding, terminal-use, and computer-use benchmarks, IHR-executed NLAHs achieve comparable task outcomes to code and prompted realizations, while exposing much shorter static harness policies. Module ablations further show that explicit harness modules are analyzable. These results suggest that agent harnesses can be turned from incidental glue around models into scientific representation objects.
Figures & tables
Figure 1: Three ways to control an agent run. The spectrum ranges from restrictive harnessing to no external harness, or self-harnessing. Code harnesses impose hard external control on a model through program logic. NLAH+IHR, the design point studied in this paper, moves the harness policy into readable natural language while a shared runtime executes that policy through child-agent calls. Self-harnessing is a possible future design in which a controller model directly harnesses other models without any external harness.
Figure 2: NLAH+IHR framework. A native code harness mixes policy and mechanism inside controller code. NLAH+IHR separates them: the NLAH stores readable harness policy, IHR provides shared execution semantics, and scripts or adapters handle exact operations such as tools, tests, parsers, and validators.
| Benchmark | Harness | Type | Perf. | LLM Calls | Tool Calls | Pr. Tok. | Comp. Tok. | Run time (min) |
|---|---|---|---|---|---|---|---|---|
| SWE Verified | Live-SWE | Code | 67.00 | 23.30 | 17.70 | 283.60k | 3.50k | 28.90 |
| Prompt | 77.00 | 36.40 | 48.00 | 2.20M | 27.50k | 5.70 | ||
| NLAH | 73.00 | 41.00 | 63.40 | 2.20M | 32.30k | 6.10 | ||
| TB2 | MHTBA | Code | 36.00 | 223.20 | 122.90 | 10.40M | 17.50k | 19.50 |
| Prompt | 57.30 | 41.50 | 48.00 | 3.10M | 51.80k | 11.10 | ||
| NLAH | 53.90 | 56.40 | 78.00 | 4.20M | 74.80k | 13.50 |
Table 1: RQ1: NLAH execution preserves competitive task performance while exposing process costs. Perf. is the benchmark primary percentage metric. Code denotes the native code harness, Prompt denotes the same NLAH text used as ordinary instructions, and NLAH denotes IHR-executed NLAH. Token and call metrics report the observable process cost of each realization.
| Benchmark | Harness | Token | Files | ||
|---|---|---|---|---|---|
| Code | NLAH | Code | NLAH | ||
| SWE Verified | Live-SWE | 60.10k | 2.90k | 68.00 | 3.00 |
| TB2 | MHTBA | 10.50k | 0.80k | 3.00 | 1.00 |
| OSWorld | SeeAct | 47.50k | 1.40k | 5.00 | 1.00 |
Table 2: RQ1: NLAHs expose the reusable harness policy in fewer static materials. Counts include audited static NLAH files and corresponding code-harness implementation materials, not runtime prompts or generated logs.
| Benchmark | Harness | Type | Verification Signals | Prompt Contract | Tool Surface | Workflow Pres. | Stage Cov. | Ordered Workflow | Context Boundary | Model Match |
|---|---|---|---|---|---|---|---|---|---|---|
| SWE Verified | Live-SWE | Code | 3.99 | - | - | - | - | - | - | - |
| SWE Verified | Live-SWE | Prompt | 6.51 | 0.89 | 0.82 | 0.70 | 0.75 | 0.74 | 1.00 | 1.00 |
| SWE Verified | Live-SWE | NLAH | 9.89 | 0.81 | 0.87 | 0.67 | 0.82 | 0.78 | 0.76 | 0.76 |
| Terminal-Bench 2.0 | MHTBA | Code | 45.05 | - | - | - | - | - | - | - |
| Terminal-Bench 2.0 | MHTBA | Prompt | 13.18 | 1.00 | 0.81 | 0.64 | 0.57 | 0.53 | 1.00 | 0.99 |
| Terminal-Bench 2.0 | MHTBA | NLAH | 22.82 | 0.84 | 0.80 | 0.63 | 0.57 | 0.54 | 0.81 | 0.55 |
Table 3: RQ2: NLAHs preserve recognizable harness-pattern structure. Except for Verification Signals, the Code row is the reference harness and therefore does not receive one-way similarity scores.
| Benchmark | Harness | Type | Artifact Contract | Tool Call Success | Failed Tool Continuation | Cached Token Ratio | Orchestration Reliability | Information Handoff Recall |
|---|---|---|---|---|---|---|---|---|
| SWE Verified | Live-SWE | Code | 0.99 | 0.88 | 0.95 | 0.71 | NA | NA |
| SWE Verified | Live-SWE | Prompt | 0.99 | 0.93 | 0.98 | 0.96 | 1.00 | 1.00 |
| SWE Verified | Live-SWE | NLAH | 1.00 | 0.93 | 0.99 | 0.94 | 0.83 | 0.32 |
| Terminal-Bench 2.0 | MHTBA | Code | NA | 0.95 | 0.79 | 0.00 | NA | NA |
| Terminal-Bench 2.0 | MHTBA | Prompt | 1.00 | 0.92 | 1.00 | 0.96 | 0.99 | 1.00 |
| Terminal-Bench 2.0 | MHTBA | NLAH | 0.96 | 0.93 | 1.00 | 0.94 | 0.85 | 0.55 |
Table 4: RQ2: NLAHs instantiate harness-engineering mechanisms. For Prompt, Orchestration Reliability and Information Handoff Recall use direct-context variants; for NLAH, they use parent-child handoff variants.
| Setting | SWE Verified | OSWorld | ||
|---|---|---|---|---|
| Perf. | Agent Calls | Perf. | Agent Calls | |
| Basic | 73.00 | 1.10 | 44.40 | 1.08 |
| + File-backed state | 75.60 +2.60 | 1.10 0.00 | 58.30 +13.90 | 1.11 +0.03 |
| + Evidence-backed answering | 75.80 +2.80 | 1.20 +0.10 | 47.20 +2.80 | 1.06 -0.03 |
| + Verifier | 73.20 +0.20 | 2.30 +1.20 | 52.80 +8.40 | 1.42 +0.33 |
| + Self-evolution | 78.80 +5.80 | 1.20 +0.10 | 52.80 +8.40 | 1.19 +0.11 |
Table 5: RQ3: Explicit NLAH modules can be ablated under a shared runtime. Each row adds one module to a benchmark-specific Basic condition. Perf. is the main benchmark metric, and Agent Calls measures changes in execution topology. Values should be compared within the same benchmark column, not across benchmark families.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Outcome group | Count | Share | Interpretation |
|---|---|---|---|
| Resolved without timeout | 11 | 12.4% | The code artifact both solved and stopped normally. |
| Resolved with timeout | 21 | 23.6% | The verifier reward is already 1.0, but the agent loop did not terminate cleanly. |
| Failed without timeout | 12 | 13.5% | The run failed for reasons other than the global agent timeout. |
| Failed with timeout | 45 | 50.6% | The dominant failure mode is timeout after a long control loop. |
Table 6: MHTBA code-artifact timeout diagnostics on TB2 under GPT.
| Slice | N | Timeouts | Mean episodes | Mean input tokens | Mean no-tool warnings |
|---|---|---|---|---|---|
| Code fail, Prompt success | 26 | 25 | 316.5 | 16.3M | 146.9 |
| Code fail, NLAH success | 24 | 22 | 285.2 | 14.7M | 124.3 |
| Code fail, both Prompt and NLAH success | 19 | 19 | 318.3 | 15.4M | 146.4 |
| All three settings success | 23 | 16 | 174.3 | 5.2M | 79.1 |
Table 7: MHTBA code-artifact trajectory diagnostics for cross-condition disagreements.
| Aspect | Core question |
|---|---|
| Agent loop | How does the agent observe, plan, act, validate, update state, retry, and stop over a long run? |
| Tool design and documentation | Which tools are exposed, how are their schemas and descriptions written, and how are tool errors returned? |
| Context engineering | What should each call see, what should be loaded just in time, and when should context be compressed or refreshed? |
| Filesystem and workspace | Where do task inputs, intermediate files, scratch files, and final artifacts live? |
| Memory and state | What information persists across steps, retries, context compression, or future runs, and what is the authoritative carrier? |
| Validation and stopping | What evidence is required before success, failure, retry, or budget exhaustion can be declared? |
Table 8: Harness-engineering aspects and their core design questions.
| Layer | Owner | Responsibility |
|---|---|---|
| Base runtime code | Code | Model APIs, LiteLLM routing, tool schemas, bash execution, timeouts, event streams, message history, context-token estimation, and run state; code also carries what must be precise, reproducible, safe, fast, or dependent on external systems, including tool calls, file execution, sandboxing, parsers, and evaluators. |
| Runtime policy / charter | Fixed NL | IHR interpretation semantics, including parent-child boundaries, the meaning of an agent call, artifact and STATE_ROOT semantics, completion gates, and audit requirements; it also carries shared interpretation rules, such as the rule that a parent orchestrator does not do substantive task work while executor children perform the task. |
| NLAH | Replaceable NL | The concrete harness roles, stages, loops, validation policy, recovery policy, state strategy, delegation policy, and module composition; natural language should carry roles, stages, strategies, when to validate, how to recover, what evidence is sufficient, and which modules can be ablated. |
| Scripts / adapters | Code hook | Tests, validators, parsers, benchmark wrappers, artifact post-processing, and any operation that requires exact execution; script hooks are task-family-specific precise operations that should not rely on natural-language interpretation and are executed by agents from the filesystem through bash . |
| Model internals | Not NLAH | Constrained decoding, logit bias, sampling, and model-internal reasoning mechanisms, unless a runtime or provider exposes them as code-level hooks. |
Table 9: Natural-language/code boundary in an IHR+NLAH system.
| Aspect | Base-agent code | Runtime policy | NLAH | Scripts/adapters |
|---|---|---|---|---|
| Agent loop | Minimal model-tool loop and child launch primitive | Parent orchestrator role, child boundary, stop discipline | Task-family stages, branching, role responsibilities | Optional loop drivers for fixed protocols |
| Tool design and documentation | Tool schemas, dispatch, timeout, error capture | Shared rules for unavailable tools and tool-result handling | When and why to use each tool class | Tool wrappers and capability adapters |
| Context engineering | Message history and context compression mechanism | Forking semantics, fresh-child semantics, handoff discipline | What to pass, reopen, summarize, or keep private | Retrieval or summarization helpers |
| Filesystem and workspace | Workspace access and run-state locations | Separation between runtime state and final artifacts | Required paths, artifact ownership, overwrite rules | Artifact processors and path normalizers |
| Memory and state | Minimal persisted run parameters, messages, and events | When durable state is required for audit or reuse | Task histories, manifests, memory files, state update rules | State serializers and merge scripts |
| Validation and stopping | Ability to run commands and collect outputs | Contract-first completion and failure reporting | Acceptance gates, verifier roles, evidence requirements | Tests, validators, parsers, graders |
Table 10: Mapping code-harness aspects to IHR+NLAH carriers.
More in this topic
LLM Agent Harnesses
LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.
See all 381 papers