cs.AIOct 23, 2026
SaveTrial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
Organizations: Institute of Automation, CAS · Zhongguancun Academy · Institute of Information Engineering, CAS · Tsinghua University ·
Abstract
LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.
Figures & tables
Figure 1: Background and motivation. Agent performance depends on both the model and its harness, but task outcomes alone cannot distinguish model errors from harness shortfalls. TraceTrial-Bench uses native execution traces, artifacts, and environment state to assess agent performance and harness reliability separately and support evidence-based failure attribution.
Figure 2: TraceTrial-Bench overview. Mechanism-oriented tasks exercise native harnesses; trajectory tracing and multi-source evidence support separate Agent and Harness scores and evidence-based failure attribution. The figure’s evidence-extraction stage corresponds to the MEA module.
Figure 3: TraceTrial-Bench system boundary and implemented pipeline. The native harness is the SUT. Dashed arrows denote evidence acquisition; solid arrows denote task execution or subsequent analysis. The evaluator consumes normalized evidence only after run-identity and apparatus checks.
Figure 4: Taxonomy and task coverage of TraceTrial-Bench. (Left) Distribution of 40 mechanism-level reliability properties across seven Harness components. The inner ring shows each component’s share, and the outer ring lists individual properties, with each property assigned equal weight. (Right) Scope of each dimension and its task coverage in the 95-task evaluation population.
| Contract element | Fixed before execution | Evaluation role |
| Task and pressure | Prompt, fixtures, mechanism trigger, and resource limits | Exercise a declared native behavior within a realistic task. |
| Agent score sheet | Required actions and final deliverables | Check task completion from artifacts and authoritative state. |
| Harness score sheet | Native obligation, observation boundary, and evidence selector | Check faithful information delivery and execution behavior. |
| Oracle and provenance | Private state, workspace baseline, and run identity | Separate environment truth from model-facing claims. |
Table 1: Task contracts and evidence boundaries. Each task combines a Harness obligation with an Agent deliverable and an independent truth anchor. The detailed catalog is in Appendix .
| Family | Representative event types | Principal normalized content |
| Lifecycle | run_started , run_finished , run_failed | Run identity, native session, exit state, timeout, and trace-completeness boundary. |
| Model surface | model_input , tool_inventory , assistant_message , assistant_delta | Messages or declarations observed at a supported request boundary, assistant content, model identity, and finish reason. |
| Action/ observation | tool | Correlated call ID, tool name, arguments, model-facing result content, error/status fields, timing, and the native call/result record. |
| Stateful control | context_compaction , subagent_lifecycle , policy_event | Compaction boundary, parent/child identity, completion delivery, approval, or policy transitions when exposed natively. |
| Accounting/ archive | token_usage , native_session_archive , raw_stdout , raw_stderr | Usage, archive provenance and digest, or raw material that could not be normalized safely. |
Table 2: Canonical event families used by the current tracer. Optional fields are emitted only when the native source supports them.
Figure 5: Cross-boundary reconstruction of a tool action. The model-facing result, native record, external execution state, and normalized event are related but not interchangeable. Solid arrows show execution; dashed arrows terminate in separately sourced, identity-aligned evidence.
| Source | Collected by | Primary supported claim | Corroborating source for |
| Simulator ledger and final state | Runner-owned private service | Exact requests, issued values, irreversible state, and oracle truth | Model-visible presentation of the corresponding result |
| Workspace baseline and changed artifacts | Persisted scoring inputs and post-run workspace | Delivered file existence, structure, content, and unauthorized changes | Trajectory and tool evidence explaining their production |
| Native normalized trajectory | Harness-specific adapter | Model-facing calls/results when the native source exposes them, lifecycle, actor lineage, and final messages | External process or service truth |
| Provider-request evidence | Runner-side recorder | Successful structured tool declarations, request status, timing, and usage | Native delivery and subsequent environment effects |
| Execution/security probes | Runner-owned post-tool observers | Process disposition, timeout, path, approval, and actor observations independent of prose reports | Model visibility through a correlated native result |
| Agent claims | Final messages and output artifacts | What the agent asserted or delivered | Harness behavior and external truth through independent evidence |
Table 3: Evidence sources and their implemented epistemic scope.
Figure 6: Implemented evidence-packet construction. Selection is deterministic and criterion-specific, while the complete archive remains available for audit and replay.
| State | Scope | Judge-side meaning | Final binary treatment |
| 0 Unsupported | Agent or Harness | The criterion is violated, or the selected evidence does not establish satisfaction. | Finalize as fail (0). |
| 1 Supported | Agent or Harness | The selected evidence establishes that the criterion is satisfied. | Finalize as pass (1). |
| 2 Not exercised | Harness rubric only | The native mechanism is absent, or the run did not reach the relevant execution path. | Resolve from the Agent outcome: pass when all Agent criteria pass, and fail otherwise; retain state 2. |
| 3 Malfunction | Harness rubric only | The mechanism exists and was reached, but the Harness itself malfunctioned. | Finalize as Harness fail (0), independent of the Agent outcome; retain state 3. |
Table 4: Judge states and their binary resolution. Deterministic criteria use states 0–1, while semantic Harness criteria may additionally use states 2–3. Every finalized criterion remains in the layer denominator.
Figure 7: Admission and layered-evaluation order. Agent results are completed before Harness semantic judging, and the two layers retain distinct evidence packets and verdict vectors.
| Order | First matching run state | Stored label | Operational meaning |
| 1 | Apparatus admission fails ( ). | inconclusive_device | Exclude the run from Harness–Agent attribution. |
| 2 | Either layer has no measured pass/fail criterion. | inconclusive_unmeasured | At least one evaluation layer is unmeasured. |
| 3 | Harness zero and outcome issue, with . | harness_caused | A candidate Harness-to-Agent mechanism is activated; validation is reported separately through . |
| 4 | Harness zero and outcome issue, with . | agent_caused | The outcome issue is unlinked; concurrent Harness zeros remain recorded on the Harness axis. |
| 5 | Harness zero, but all Agent criteria pass and no bypass is recorded. | harness_defect_absorbed | The Agent delivery is complete despite an operational Harness zero; determines defect confirmation. |
| 6 | No Harness zero remains, but an outcome issue is present. | agent_caused | The Agent-side zero or measurement bypass is assigned to the Agent axis. |
Table 5: Runtime-label decision table. Rows are evaluated from top to bottom, and the first matching row determines the stored label. A “Harness zero” means ; an “outcome issue” means . The labels summarize operational evidence states rather than validated causal conclusions.
| Agent | Harness | |||||
| Harness | Pass | Score | Micro | Macro | Clean | Overall |
| Claude Code | 44.0 | 75.9 | 71.5 | 71.2 | 9.1 | 73.5 |
| Codex | 35.8 | 70.7 | 60.2 | 59.8 | 2.8 | 65.2 |
| DeepSeek Harness | 40.0 | 73.3 | 69.5 | 68.8 | 9.8 | 71.0 |
| OpenClaw | 38.2 | 71.7 | 71.9 | 72.0 | 11.7 | 71.8 |
| All harnesses | 39.5 | 72.9 | 68.3 | 67.9 | 8.4 | 70.4 |
Table 6: Aggregate results (%). All harnesses use 855 matched model–task cells. Overall averages unrounded Agent Score and Harness Macro equally, matching the first-page plots. Agent Pass requires all Agent criteria to pass, while Clean requires all criteria in both layers to pass. Score and Macro average per-run fractions, whereas Micro pools Harness items. Full model-level results appear in Table 5.3 .
More in this topic
LLM Agent Harnesses
LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.
See all 381 papers