Trace-to-Architecture: Capability Bottleneck Diagnosis for Harness Revision

Wenfeng Wang; Jianrui Shi; Mingang Guo; Jiayuan Ma; Yuqi Lu; Jiajun Sun; Guojiang Zhao; Jing Li

cs.AIOct 12, 2026

Trace-to-Architecture: Capability Bottleneck Diagnosis for Harness Revision

Authors: Wenfeng Wang, Jianrui Shi, Mingang Guo, Jiayuan Ma, Yuqi Lu, Jiajun Sun, Guojiang Zhao, Jing Li

Organizations: University of Pennsylvania, Philadelphia, USA · Harbin Institute of Technology, Shenzhen, China · DP Technology, Beijing, China

Abstract

Improving an LLM agent requires knowing what to change when it fails. Similar failures can require different fixes, and resolving one problem can expose another. We introduce Trace-to-Architecture (T2A), a framework that uses execution trajectories to determine what to change in an agent's harness and when those changes apply. A development agent diagnoses failures using execution evidence from successful and failed tasks, revises the corresponding harness components, and then checks whether target tasks improve, selected successful tasks remain successful, and the edits are actually triggered. New execution evidence guides subsequent diagnosis and revision. Across ALFWorld, ScienceWorld, and AppWorld, T2A achieves the highest aggregate score for all three evaluated task agents, exceeding their respective strongest baselines by 6.2--30.3 points. We also integrate T2A's diagnosis and validation procedures into Meta-Harness, improving the resulting harnesses' mean ScienceWorld score by 21.6 points from the same starting harness.

Figures & tables

Figure 1: T2A alternates between trajectory collection (top) and diagnosis, revision, and validation (bottom). In the example, execution evidence identifies a skill card that is absent from the model input. Adjusting its invocation conditions improves the target task, but regressions on related tasks lead to rejection and further revision.

Benchmark	Baselines						Ours	$\boldsymbol{\Delta}$
Benchmark	Minimal prompt	ReAct	SkillNet	GEPA	WorldEvolver	Meta-Harness	T2A	$\boldsymbol{\Delta}$
Qwen3.5-27B
ScienceWorld	$23.1\,{\scriptstyle\pm 1.6}$	$11.0\,{\scriptstyle\pm 1.1}$	$31.1\,{\scriptstyle\pm 1.3}$	$58.6\,{\scriptstyle\pm 0.8}$	$29.5\,{\scriptstyle\pm 1.8}$	$63.7\,{\scriptstyle\pm 0.5}$	$\mathbf{87.7}\,{\scriptstyle\pm 0.1}$	$+24.0$
AppWorld Normal	$58.9\,{\scriptstyle\pm 2.1}$	$33.3\,{\scriptstyle\pm 2.7}$	–	$86.1\,{\scriptstyle\pm 1.7}$	$57.1\,{\scriptstyle\pm 1.8}$	$79.8\,{\scriptstyle\pm 1.2}$	$\mathbf{90.1}\,{\scriptstyle\pm 1.2}$	$+4.0$
AppWorld Challenge	$51.9\,{\scriptstyle\pm 0.5}$	$18.9\,{\scriptstyle\pm 1.1}$	–	$72.2\,{\scriptstyle\pm 1.6}$	$47.5\,{\scriptstyle\pm 0.7}$	$65.9\,{\scriptstyle\pm 0.7}$	$\mathbf{74.0}\,{\scriptstyle\pm 1.5}$	$+1.8$
ALFWorld Seen	$65.2\,{\scriptstyle\pm 1.8}$	$67.9\,{\scriptstyle\pm 1.4}$	$89.3\,{\scriptstyle\pm 0.7}$	$90.0\,{\scriptstyle\pm 0.7}$	$64.3\,{\scriptstyle\pm 1.4}$	$91.7\,{\scriptstyle\pm 0.8}$	$\mathbf{97.4}\,{\scriptstyle\pm 0.4}$	$+5.7$

Table 1: Test results (mean

\pm

sample standard deviation over three evaluations). Bold marks the highest mean in each row. Average weights the three environments equally, averaging the two splits within AppWorld and ALFWorld.

\Delta

is T2A’s difference from the highest baseline mean in the same row and model block. SkillNet has no AppWorld results or Average because its public release provides no AppWorld skill library or evaluation setup.

Figure 2: ScienceWorld fatal-failure rates, averaged over three evaluations. Lower is better; colors distinguish the three task models.

Figure 3: Failure categories for initial (minimal-prompt), intermediate, and final T2A harnesses, from one evaluation per harness. Cells show percentages of all test tasks in each environment;

<1

denotes less than 1%. Definitions and counting rules are in Appendix D.5 .

Removed	Mean	$\boldsymbol{\Delta}$
None (full)	85.2	—
Cards	65.0	$-20.2$
Check	70.4	$-14.8$
Memory	70.9	$-14.3$
Cards + check	52.3	$-32.9$
Cards + memory	46.5	$-38.7$

Table 2: Individual and combined component ablations on ScienceWorld.

\Delta

is the task-paired score difference from the full system. Evaluation and implementation details are in Appendix D.3 .

Figure 4: ScienceWorld development comparisons with DeepSeek V4.1 Flash. Mean scores appear above zero and fatal-failure rates (%) below. Results are averaged over three independent development runs; error bars show sample standard deviations of scores. Starting harnesses are evaluated once. MH: Meta-Harness.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Diagnostic evidence	How edits are organized	Validation and selection
Self-Harness ( Zhang et al., 2026a )	Failed trajectories clustered by verifier-grounded failure mechanisms.	Bounded component edits with audit records and passing behaviors to preserve.	Held-in and held-out pass counts must not decrease; at least one must improve.
AHE ( Lin et al., 2026a )	Layered trajectory reports and manifests linking evidence, causes, and predicted effects.	Component edits in a decoupled harness workspace.	Next-round attribution checks scores for expected fixes and regressions and supports rollback.
HarnessX ( Chen et al., 2026c )	Task briefs and a cross-task outcome overview with edit history.	Edits to typed processors, recorded in change manifests.	A predefined procedure for review, checks, and regression gates.
HarnessFix ( Chen et al., 2026a )	Trace intermediate representation links responsible steps to harness-layer flaw records.	Repair operators instantiated with file and behavior constraints.	Repair-scope checks, target-flaw reduction, and regression validation.
EvolveNet ( Nie et al., 2026 )	Client-specific experience and evidence of gains and regressions.	Aggregate local program adaptations into shared or domain-specific changes.	Commit the integrated harness only if validation gains at least match regressions.
Meta-Harness ( Lee et al., 2026 )	The proposing agent reads historical code, scores, and execution trajectories.	Free-form code changes chosen by the proposer.	Search-set evaluation and a performance–cost Pareto frontier.

Appendix

Table 3: How related methods diagnose failures, organize edits, and validate candidates.

Function	ScienceWorld	AppWorld	ALFWorld
Structured records	Present	Present	Present
Within-episode memory	Interaction history; confirmed object locations	Interaction history	Recent history
Experience retrieval	Empty card library	None	Empty card library
Action processing	On	None	None
Action checks / execution repair	Off	None	None

Appendix

Table 4: Support functions in the initial harnesses. Off denotes code that is present but disabled, and none denotes an absent component.

Component	Full	Removed
Cards	4	28
Check	0	30
Memory	14	48

Appendix

Table 5: Tasks containing the target error in the full system and after individual component removal. Each row uses its component-specific error definition described in this section.

Task situation	Old trajectory	Revised trajectory	What the trajectories establish
Prepare an intermediate color for tertiary mixing	Repeated kitchen search without finding materials; final score 0	Enter the art studio, mix blue/yellow paint; environment confirms green at 23 points; then select the wooden cup as the target and end at $-100$	Material location and preparation improve, but target reference remains faulty; lower final score obscures earlier progress
Compare inclined-plane angles	Search/navigation without a descent experiment; final score 10	Complete the first descent and read time; while preparing the second, use focus on the stopwatch as a reset and end at $-100$	First experiment progresses, but using focus to reset the stopwatch ends the task
Infer dominant/recessive plant traits	Sow seeds but fail to obtain water effectively; plants do not grow; final score 16	Grow two parent plants to flowering, but answer from parental leaf shape instead of offspring evidence; end at $-100$	Care operations progress, but genetic evidence remains insufficient

Appendix

Table 6: Historical same-task cases with local progress and subsequent fatal failure, evaluated under the execution limits in effect when each harness version was evaluated.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 389 papers

From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws
Jun 4, 2026Mengzhuo Chen, Junjie Wang, Zhe Liu +3Agent Failure Analysis Self-Improving Agents
State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1
Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
Oct 23, 2026Zhiyong Cao, Hengjie Zhu, Yongxin Wang +20LLM Agent Harnesses LLM Agent Reliability
Institute of Automation, CAS · Zhongguancun Academy · Institute of Information Engineering, CAS +2
Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
May 20, 2026Akshay Manglik, Vijay S. Kalmath, Jason Qin +6Multi-Agent LLM Systems LLM Agent Evaluation
Scale AI

Jun 4, 2026J/K move · Enter open · S save

From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang+2

State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China+1

Agent Failure Analysis Self-Improving Agents LLM Agent Harnesses LLM Agent Reliability

Open paper AskPDF arXiv

LLM agents increasingly rely on agent harness: the runtime infrastructure around the base model that defines execution environments, tool interfaces, context, lifecycle orchestration, observability, verification, and governance. Existing self-improving agents and automatic harness evolution methods mainly improve agents through runtime supervision, prompt optimization, workflow search, or harness modification based on final outcomes. However, they often fail to diagnose where the responsible evidence lies in failed trajectories and which harness implementation mechanism causes the unreliable behavior, resulting in broad, indirect, or poorly scoped changes. This paper proposes HarnessFix, a trace-grounded and diagnosis-driven framework for repairing agent harnesses. HarnessFix compiles raw execution traces and harness artifacts into a Harness-aware Trace Intermediate Representation (HTIR), which normalizes fragmented trajectory evidence, captures step-level data-flow and control-flow relations, and aligns runtime steps with the harness artifacts that shape their behavior. It then attributes failures to responsible steps and harness artifacts, and consolidates recurring diagnoses into repair-oriented flaw records. Finally, HarnessFix maps these records to scoped repair operators, generates patches under flaw-specific repair specifications, and accepts them through regression-aware validation. We evaluate HarnessFix on four popular benchmarks, and results show that it improves the performance over the initial harnesses by 6.3% to 18.4%, significantly outperforming human-designed and self-evolution baselines. HarnessFix highlights the value of treating failed trajectories not only as feedback signals, but also as structured evidence for diagnosing and repairing the harness mechanisms behind agent failures.

Abstract

LLM Agent Harnesses

Explore similar work

From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents