Large language model (LLM) agents increasingly interact with tools and environments, yet plausible model decisions do not always translate into successful execution due to tool-call mismatches, interrupted workflows, and unsupported completion claims. We present EvoX, a Rust-based, multi-model agent runtime guided by The Second Policy, which views the harness as an Operational Policy complementing the foundation model's Semantic Policy. EvoX combines model-facing guidance with deterministic runtime control through three complementary mechanisms: Action Realization, Execution Continuity, and Outcome Grounding. To evaluate harnesses without suppressing their native behaviors, we introduce Contrast, a controlled and auditable framework integrating explicit fairness constraints, provider-bound request verification, and evidence-based adjudication. Across five benchmarks, six foundation models, and six agent harnesses, EvoX achieves the highest numerical success rate of 56.2% over 691 common task--model pairs, exceeding the average of the other five harnesses by approximately 3.0 percentage points. Its larger advantages on GLM-5.3 (+10.7 points) and Kimi K3 (+7.9 points) highlight the model-dependent effectiveness of harness design and the potential for cross-provider adaptability. Achieved without memory, evolution, or extensions, these results establish EvoX as a competitive multi-model agent runtime and underscore the importance of harness-level operational control in realizing foundation-model capabilities.
Figures & tables
Figure 1: Architecture and runtime design of EvoX . The system integrates a model-driven Semantic Policy with a harness-level Operational Policy to coordinate decisions and environment interactions. The operational policy comprises three complementary mechanisms: Action Realization mediates executable tool calls, Execution Continuity coordinates recovery and runtime control, and Outcome Grounding aligns reported outcomes with observed execution. The runtime supports diverse tools and optional persistent adaptation through long-term memory and GEP-based experience evolution. The right panel illustrates the conceptual distinction between a minimal execution loop and explicit harness-level operational control.
Figure 2: Overview of Contrast for controlled and auditable harness evaluation. A fairness contract distinguishes locked, shared, and harness-native conditions, preserving native execution behaviors while controlling external evaluation factors. Versioned tasks undergo qualification and integrity checks before execution through native harnesses in controlled environments. A model bridge mediates model requests, enforces applicable configurations, and audits provider-bound request parameters. Execution traces, model-call records, verifier results, and runtime diagnostics provide evidence for failure attribution, validity decisions, and auditable result selection. Only valid, scored task–model combinations enter paired statistical comparisons, while excluded or quarantined records remain available for audit and disclosure.
Figure 3: Overall performance of six agent harnesses on 691 common task–model pairs across five benchmarks. Bars indicate task success rates, with EvoX highlighted in blue. The dashed line represents the mean success rate of the other five harnesses (53.1%).
Figure 4: Harness–model performance across six foundation models. Each cell reports the task success rate (%) of a harness–model combination, with outlined cells indicating the highest performance for each model. The overall column summarizes performance across 691 common task–model pairs. The bottom row reports the performance difference between EvoX and the mean of the other five harnesses, in percentage points.
Benchmark
EvoX
Pi
Claude Code
Hermes
Prime
Codex
Δ (pp)
Aider Polyglot
64.7
61.7
62.3
63.5
61.1
55.1
+4.0
GAIA
86.7
85.7
82.9
85.7
85.7
78.1
+3.0
Harbor Index
27.3
25.9
31.5
24.5
23.1
27.3
+0.8
SWE-Live
48.4
49.2
49.2
43.7
35.7
37.3
+5.4
Terminal-Bench 2.1
57.3
60.7
55.3
59.3
50.0
54.0
+1.5
Table 1: Performance across five benchmark families. Each entry reports task success rate (%) on common task–model pairs, with n denoting the number of pairs per benchmark. Δ indicates the difference between EvoX and the mean of the other five harnesses, in percentage points. Best results are highlighted in bold.
Normal
Other outcomes
Rate (%)
Harness
Pass
Fail
Exit
TO-F
TO-P
Cap-P
Cap-F
Other
Normal
Pass ∣ Normal
EvoX
373
134
117
49
4
12
2
0
73.4
73.6
Pi
365
187
5
107
6
18
2
1
79.9
66.1
Claude Code
344
161
111
30
5
37
3
0
73.1
68.1
Hermes
361
150
159
2
0
14
5
0
74.0
70.6
Prime Agent
330
152
119
68
4
15
3
0
69.8
68.5
Table 2: Execution outcome profiles across 691 common task–model pairs. Normal completion excludes nonzero exits, timeouts, and cost-cap interruptions. Exit denotes a nonzero process exit; TO and Cap denote timeout and cost-cap termination, respectively, with P/F indicating passing or failing verifier outcomes. A passing verifier result after a timeout is scored as unsuccessful under the evaluation protocol. The final columns report the normal-completion rate and verifier success rate conditional on normal completion.
Figure 5: Model-composition sensitivity across six agent harnesses. The first column reports each harness’s performance advantage over the mean of the other five harnesses on the full evaluation set. Each remaining entry shows the change in this relative advantage after excluding the indicated foundation model, measured in percentage points. Negative values indicate reduced relative competitiveness, while positive values indicate improvement.
AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.
Long-horizon LLM agents increasingly rely on external execution support to maintain state, track progress, recover from failures, and reuse experience across extended interactions. Yet existing harnesses and their use are often tailored to environments and controlled through prompts, heuristics, or system-specific rules, making agent and harness coordination difficult to jointly optimize. We introduce EvoHarness-RL, a unified framework that separates environment-specific harness implementations from a shared policy-facing interface. EvoHarness-RL organizes external support into a Belief, Progress, and Experience (BPE) workspace and exposes four compact harness actions for accessing and updating this state. We first instantiate BPE as an inference-time scaffold and then make harness coordination learnable through supervised initialization followed by cost-aware GRPO. Across heterogeneous long-horizon tasks, EvoHarness-Base improves the average success rate of frontier models by 10.0 percentage points, while EvoHarness-RL outperforms the strongest open-source baseline by 8.5 percentage points, together with higher RL rollout efficiency and stronger generalization to unseen tasks. Our analyses show that training gradually shifts agents from frequent scaffold use toward selective, environment-dependent harness access as the policy becomes more capable, while the external workspace continues to evolve and refine itself to better support task execution and generalization. Together, these results show that long-horizon agents benefit not only from external scaffolding itself, but also from learning how and when to coordinate with external support as part of a cost-aware policy.