cs.AIOct 12, 2026

Trace-to-Architecture: Capability Bottleneck Diagnosis for Harness Revision

Authors: Wenfeng Wang, Jianrui Shi, Mingang Guo, Jiayuan Ma, Yuqi Lu, Jiajun Sun, Guojiang Zhao, Jing Li

Organizations: University of Pennsylvania, Philadelphia, USA · Harbin Institute of Technology, Shenzhen, China · DP Technology, Beijing, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Improving an LLM agent requires knowing what to change when it fails. Similar failures can require different fixes, and resolving one problem can expose another. We introduce Trace-to-Architecture (T2A), a framework that uses execution trajectories to determine what to change in an agent's harness and when those changes apply. A development agent diagnoses failures using execution evidence from successful and failed tasks, revises the corresponding harness components, and then checks whether target tasks improve, selected successful tasks remain successful, and the edits are actually triggered. New execution evidence guides subsequent diagnosis and revision. Across ALFWorld, ScienceWorld, and AppWorld, T2A achieves the highest aggregate score for all three evaluated task agents, exceeding their respective strongest baselines by 6.2--30.3 points. We also integrate T2A's diagnosis and validation procedures into Meta-Harness, improving the resulting harnesses' mean ScienceWorld score by 21.6 points from the same starting harness.

Figures & tables

Figure 1: T2A alternates between trajectory collection (top) and diagnosis, revision, and validation (bottom). In the example, execution evidence identifies a skill card that is absent from the model input. Adjusting its invocation conditions improves the target task, but regressions on related tasks lead to rejection and further revision.
BenchmarkBaselinesOursΔ\boldsymbol{\Delta}
Minimal promptReActSkillNetGEPAWorldEvolverMeta-HarnessT2A
Qwen3.5-27B
ScienceWorld23.1 ±1.623.1\,{\scriptstyle\pm 1.6}11.0 ±1.111.0\,{\scriptstyle\pm 1.1}31.1 ±1.331.1\,{\scriptstyle\pm 1.3}58.6 ±0.858.6\,{\scriptstyle\pm 0.8}29.5 ±1.829.5\,{\scriptstyle\pm 1.8}63.7 ±0.563.7\,{\scriptstyle\pm 0.5}87.7 ±0.1\mathbf{87.7}\,{\scriptstyle\pm 0.1}+24.0+24.0
AppWorld Normal58.9 ±2.158.9\,{\scriptstyle\pm 2.1}33.3 ±2.733.3\,{\scriptstyle\pm 2.7}–86.1 ±1.786.1\,{\scriptstyle\pm 1.7}57.1 ±1.857.1\,{\scriptstyle\pm 1.8}79.8 ±1.279.8\,{\scriptstyle\pm 1.2}90.1 ±1.2\mathbf{90.1}\,{\scriptstyle\pm 1.2}+4.0+4.0
AppWorld Challenge51.9 ±0.551.9\,{\scriptstyle\pm 0.5}18.9 ±1.118.9\,{\scriptstyle\pm 1.1}–72.2 ±1.672.2\,{\scriptstyle\pm 1.6}47.5 ±0.747.5\,{\scriptstyle\pm 0.7}65.9 ±0.765.9\,{\scriptstyle\pm 0.7}74.0 ±1.5\mathbf{74.0}\,{\scriptstyle\pm 1.5}+1.8+1.8
ALFWorld Seen65.2 ±1.865.2\,{\scriptstyle\pm 1.8}67.9 ±1.467.9\,{\scriptstyle\pm 1.4}89.3 ±0.789.3\,{\scriptstyle\pm 0.7}90.0 ±0.790.0\,{\scriptstyle\pm 0.7}64.3 ±1.464.3\,{\scriptstyle\pm 1.4}91.7 ±0.891.7\,{\scriptstyle\pm 0.8}97.4 ±0.4\mathbf{97.4}\,{\scriptstyle\pm 0.4}+5.7+5.7
Table 1: Test results (mean ±\pm sample standard deviation over three evaluations). Bold marks the highest mean in each row. Average weights the three environments equally, averaging the two splits within AppWorld and ALFWorld. Δ\Delta is T2A’s difference from the highest baseline mean in the same row and model block. SkillNet has no AppWorld results or Average because its public release provides no AppWorld skill library or evaluation setup.
Figure 2: ScienceWorld fatal-failure rates, averaged over three evaluations. Lower is better; colors distinguish the three task models.
Figure 3: Failure categories for initial (minimal-prompt), intermediate, and final T2A harnesses, from one evaluation per harness. Cells show percentages of all test tasks in each environment; <1<1 denotes less than 1%. Definitions and counting rules are in Appendix D.5 .
RemovedMeanΔ\boldsymbol{\Delta}
None (full)85.2—
Cards65.0−20.2-20.2
Check70.4−14.8-14.8
Memory70.9−14.3-14.3
Cards + check52.3−32.9-32.9
Cards + memory46.5−38.7-38.7
Table 2: Individual and combined component ablations on ScienceWorld. Δ\Delta is the task-paired score difference from the full system. Evaluation and implementation details are in Appendix D.3 .
Figure 4: ScienceWorld development comparisons with DeepSeek V4.1 Flash. Mean scores appear above zero and fatal-failure rates (%) below. Results are averaged over three independent development runs; error bars show sample standard deviations of scores. Starting harnesses are evaluated once. MH: Meta-Harness.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

MethodDiagnostic evidenceHow edits are organizedValidation and selection
Self-Harness ( Zhang et al., 2026a )Failed trajectories clustered by verifier-grounded failure mechanisms.Bounded component edits with audit records and passing behaviors to preserve.Held-in and held-out pass counts must not decrease; at least one must improve.
AHE ( Lin et al., 2026a )Layered trajectory reports and manifests linking evidence, causes, and predicted effects.Component edits in a decoupled harness workspace.Next-round attribution checks scores for expected fixes and regressions and supports rollback.
HarnessX ( Chen et al., 2026c )Task briefs and a cross-task outcome overview with edit history.Edits to typed processors, recorded in change manifests.A predefined procedure for review, checks, and regression gates.
HarnessFix ( Chen et al., 2026a )Trace intermediate representation links responsible steps to harness-layer flaw records.Repair operators instantiated with file and behavior constraints.Repair-scope checks, target-flaw reduction, and regression validation.
EvolveNet ( Nie et al., 2026 )Client-specific experience and evidence of gains and regressions.Aggregate local program adaptations into shared or domain-specific changes.Commit the integrated harness only if validation gains at least match regressions.
Meta-Harness ( Lee et al., 2026 )The proposing agent reads historical code, scores, and execution trajectories.Free-form code changes chosen by the proposer.Search-set evaluation and a performance–cost Pareto frontier.
Appendix
Table 3: How related methods diagnose failures, organize edits, and validate candidates.
FunctionScienceWorldAppWorldALFWorld
Structured recordsPresentPresentPresent
Within-episode memoryInteraction history; confirmed object locationsInteraction historyRecent history
Experience retrievalEmpty card libraryNoneEmpty card library
Action processingOnNoneNone
Action checks / execution repairOffNoneNone
Appendix
Table 4: Support functions in the initial harnesses. Off denotes code that is present but disabled, and none denotes an absent component.
ComponentFullRemoved
Cards428
Check030
Memory1448
Appendix
Table 5: Tasks containing the target error in the full system and after individual component removal. Each row uses its component-specific error definition described in this section.
Task situationOld trajectoryRevised trajectoryWhat the trajectories establish
Prepare an intermediate color for tertiary mixingRepeated kitchen search without finding materials; final score 0Enter the art studio, mix blue/yellow paint; environment confirms green at 23 points; then select the wooden cup as the target and end at −100-100Material location and preparation improve, but target reference remains faulty; lower final score obscures earlier progress
Compare inclined-plane anglesSearch/navigation without a descent experiment; final score 10Complete the first descent and read time; while preparing the second, use focus on the stopwatch as a reset and end at −100-100First experiment progresses, but using focus to reset the stopwatch ends the task
Infer dominant/recessive plant traitsSow seeds but fail to obtain water effectively; plants do not grow; final score 16Grow two parent plants to flowering, but answer from parental leaf shape instead of offspring evidence; end at −100-100Care operations progress, but genetic evidence remains insufficient
Appendix
Table 6: Historical same-task cases with local progress and subsequent fatal failure, evaluated under the execution limits in effect when each harness version was evaluated.
More in this topic

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 389 papers

Explore similar work

CardsList
Jun 4, 2026

From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

LLM agents increasingly rely on agent harness: the runtime infrastructure around the base model that defines execution environments, tool interfaces, context, lifecycle orchestration, observability, verification, and governance. Existing self-improving agents and automatic harness evolution methods mainly improve agents through runtime supervision, prompt optimization, workflow search, or harness modification based on final outcomes. However, they often fail to diagnose where the responsible evidence lies in failed trajectories and which harness implementation mechanism causes the unreliable behavior, resulting in broad, indirect, or poorly scoped changes. This paper proposes HarnessFix, a trace-grounded and diagnosis-driven framework for repairing agent harnesses. HarnessFix compiles raw execution traces and harness artifacts into a Harness-aware Trace Intermediate Representation (HTIR), which normalizes fragmented trajectory evidence, captures step-level data-flow and control-flow relations, and aligns runtime steps with the harness artifacts that shape their behavior. It then attributes failures to responsible steps and harness artifacts, and consolidates recurring diagnoses into repair-oriented flaw records. Finally, HarnessFix maps these records to scoped repair operators, generates patches under flaw-specific repair specifications, and accepts them through regression-aware validation. We evaluate HarnessFix on four popular benchmarks, and results show that it improves the performance over the initial harnesses by 6.3% to 18.4%, significantly outperforming human-designed and self-evolution baselines. HarnessFix highlights the value of treating failed trajectories not only as feedback signals, but also as structured evidence for diagnosing and repairing the harness mechanisms behind agent failures.

Mengzhuo Chen, Junjie Wang, Zhe Liu +3

State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1

Agent Failure AnalysisSelf-Improving AgentsLLM Agent HarnessesLLM Agent Reliability
AskPDF
Oct 23, 2026

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.

Zhiyong Cao, Hengjie Zhu, Yongxin Wang +20

Institute of Automation, CAS · Zhongguancun Academy · Institute of Information Engineering, CAS +2

LLM Agent HarnessesLLM Agent ReliabilityProcess-Based Agent EvaluationFailure Attribution for LLM Agents
AskPDF
May 20, 2026

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.

Akshay Manglik, Vijay S. Kalmath, Jason Qin +6

Scale AI

Multi-Agent LLM SystemsLLM Agent EvaluationLLM Agent ReliabilityReasoning Trace Analysis
AskPDF