DAAF: From Failure Localization to Editable System Assets in LLM Agents

Xiaoyang Yuan; Qi Liu; Yubin Ruan; Xinyi Mou; Zhuomeng Zhang; Wenjin Wang; Hanying Jiao; Di Wu

cs.AISep 26, 2026

DAAF: From Failure Localization to Editable System Assets in LLM Agents

Authors: Xiaoyang Yuan, Qi Liu, Yubin Ruan, Xinyi Mou, Zhuomeng Zhang, Wenjin Wang, Hanying Jiao, Di Wu,

Organizations: Tencent

Abstract

Deployed LLM agents increasingly rely on persistent, versioned system assets such as routing rules, knowledge segments, prompt instructions, and reusable skills. Failure-localization methods can identify where an error manifests in an agent or execution trace, but repair requires a different decision: which editable system asset should be changed, and is that change expected to improve the task outcome? We study this gap through component-attribute failure attribution, where diagnosis targets versioned, addressable items rather than execution locations. We propose the Detection-Aware Attribution Framework (DAAF), which learns the effects of valid attribute replacements and amortizes this intervention evidence into deployment-time diagnosis. DAAF combines sparse and noisy failure signals to decide whether intervention is warranted, learns component-type-conditioned replacement effects from controlled replays evaluated by executable task outcomes, and shares supervision across requests with compatible intervention responses. At diagnosis time, DAAF uses only the observed execution, registered candidates, and available failure signals; it requires neither counterfactual replay nor task reward and returns no_change, a repair target, or an unresolved decision when evidence is insufficient. On held-out tau^2-bench Telecom tasks, DAAF achieves 80.72% attribute Hit@1, recovers 62.65% of failed executions while limiting clean-task regression to 3.23%, and reaches 71.93% overall task success. These results show that intervention-grounded attribute attribution can connect failure localization to executable system repair.

Figures & tables

Figure 1: Overview of DAAF. At diagnosis time (solid arrows), MSWSD estimates whether intervention is warranted and CTCA predicts signed effects for registered edits; their outputs determine no_change , a single repair, or an unresolved decision. During training (dashed arrows), weak-label targets supervise failure detection, controlled replacements evaluated by executable task outcomes supervise attribute effects, and intervention-compatible requests provide cross-request consistency.

Role	Registered attribute	Admissible replacement
Skill	Support procedure	tech_support_manual.md $\rightarrow$ tech_support_workflow.md
Knowledge	Policy evidence segment	Official policy $\rightarrow$ registered policy variant with evidence-handling guidance
Prompt	Instruction / system template	Official text $\rightarrow$ text with a fixed diagnostic or verification suffix
Harness	Parallel tool-call setting	Provider default $\rightarrow$ disabled

Table 1: Examples of registered edits in the Telecom environment. Replacements are fixed before diagnosis and are not generated by DAAF.

Method	Attribute Hit@1 $\uparrow$	Joint F1 $\uparrow$	Failure Recovery $\uparrow$	Clean Regression $\downarrow$	Overall Success $\uparrow$
No-edit Rerun	–	–	$12.05\pm 3.62$	$26.88\pm 1.86$	$28.66\pm 2.20$
Component-aware LLM	40.96	51.13	$35.74\pm 4.56$	$19.35\pm 3.23$	$47.95\pm 2.53$
LLM + MSWSD	36.14	47.62	$33.73\pm 4.82$	$24.73\pm 3.72$	$45.03\pm 2.68$
DAAF	80.72	68.02	$\mathbf{62.65\pm 3.19}$	$\mathbf{3.23\pm 3.23}$	$\mathbf{71.93\pm 1.52}$

Table 2: End-to-end results on the 114 held-out

\tau^{2}

-bench Telecom tasks. Attribute Hit@1 is evaluated on 83 natural failures; Joint F1 also includes the 31 natural successes. Repair metrics are mean

\pm

standard deviation over three matched execution repeats.

Second edit	Train Conditional Effect (pp)	Failure Recovery $\uparrow$	$\Delta$ Recovery 95% CI	Clean Regression $\downarrow$
Prompt: instruction	$+6.79$	$\mathbf{81.53\pm 2.51}$	$\mathbf{+18.88\ [10.84,26.91]}$	$\mathbf{2.15\pm 3.72}$
Prompt: system template	$+1.71$	$64.26\pm 0.70$	$+1.61\ [-4.02,7.23]$	$6.45\pm 3.23$
None (Full DAAF )	$0.00$	$62.65\pm 3.19$	reference	$3.23\pm 3.23$
Harness: parallel tool calls	$-1.72$	$62.25\pm 2.78$	$-0.40\ [-6.83,6.02]$	$4.30\pm 1.86$
Knowledge: main policy	$-5.02$	$60.64\pm 1.84$	$-2.01\ [-8.03,6.02]$	$8.60\pm 1.86$

Table 3: Transfer of training-derived conditional replacement effects. Each candidate is applied after the Full DAAF repair on held-out Telecom tasks.

\Delta

Recovery is measured in percentage points relative to Full DAAF.

	Algorithm		Handcrafted
Method	Step	Joint	Step	Joint
All-at-Once	22.22	12.70	1.72	1.72
Step-by-Step	30.16	30.16	24.14	22.41
Binary Search	0.79	0.79	12.07	12.07
AgentDebugX ( Zhu et al., 2026 )	42.06	42.06	18.97	17.24
DAAF frontend	48.02	48.02	25.86	24.14

Table 4: Native Who&When ( Zhang et al., 2025 ) accuracy (%, higher is better). Step measures exact step-level agreement; Joint requires both agent and step.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Feedback event	Failure score	Reliability
Explicit positive outcome	0.00	0.75
Explicit negative outcome	1.00	0.75
Correction	0.75	0.50
Explicit negative reaction	0.60	0.40
Abandonment	0.50	0.30

Appendix

Table 5: Frozen user-feedback mapping used by MSWSD. Larger scores indicate stronger failure evidence.

Type	Registered attribute	Replacement role
Prompt	Instruction	Adds explicit diagnostic ordering, distinguishes agent-side from user-side actions, and requests user verification before advancing.
Prompt	System template	Adds end-of-task verification and requires unresolved issues to be stated explicitly rather than inferred from tool completion.
Harness	Parallel tool calls	Changes the runtime setting from provider-default behavior to disabled parallel tool calls.
Knowledge	Main policy	Adds an evidence-handling section to the registered Telecom policy variant.
Skill	Tech-support procedure	Replaces the manual-style support description with the registered workflow-style procedure.

Appendix

Table 6: Registered Telecom attributes and replacement roles used for intervention supervision.

Setting	Value
Training seed	20260924
Base-task folds	3
Candidate budget	12
Feature / hidden dimension	64 / 32
Training epochs / warm-up	200 / 100
Learning rate	0.2

Appendix

Table 7: Final DAAF configuration used for the Telecom experiments.

Method	Attribute Hit@1 $\uparrow$	Joint F1 $\uparrow$
Uniform candidate choice	37.35	–
Component-aware LLM	40.96	51.13
LLM + CTCA	34.94	46.03
Full DAAF	80.72	68.02

Appendix

Table 8: Additional attribute-localization controls on the 83 natural Telecom failures.

Method	Failure Recovery $\uparrow$	$\Delta$ Recovery 95% CI	Clean Regression $\downarrow$
Detection-only control	$4.42\pm 2.51$	$-58.23\ [-66.27,-49.80]$	$24.73\pm 1.86$
Full DAAF	$\mathbf{62.65\pm 3.19}$	reference	$\mathbf{3.23\pm 3.23}$

Appendix

Table 9: Detection-only control under the three-repeat Telecom repair protocol.

\Delta

Recovery is measured relative to Full DAAF.

Attribute	Standalone	Composed	Change
Harness: parallel tool calls	0.62	30.85	+30.23
Knowledge: main policy	0.53	30.00	+29.47
Prompt: system template	1.25	27.42	+26.17
Prompt: instruction	0.53	21.34	+20.81
Skill: support procedure	52.20	45.35	-6.85
All attributes	11.36	31.02	$+19.66\ [14.89,24.43]$

Appendix

Table 10: Replacement activity on 4,400 controlled replay steps from 220 CTCA training tasks. Values are percentages of replacement steps that change official task reward.

	Natural cohort		Balanced development
Treatment	AUPRC	AUROC	AUPRC	AUROC
Missingness-aware ( DAAF )	75.77	56.06	66.00	73.92
Mean imputation	75.77	56.06	66.00	73.92
Zero fill	71.31	44.58	50.00	48.00

Appendix

Table 11: Failure detection under incomplete evidence (%, higher is better).

Construction	Held-out Agreement $\uparrow$	Splits Won
Intervention response ( DAAF )	62.63	10 / 10
Request similarity only	60.09	0 / 10
Random edges	58.59	0 / 10

Appendix

Table 12: Neighborhood quality under attribute hold-out, averaged across all ten splits.

Method	Online Runs / Failure $\downarrow$
No-edit Rerun	1.00
Component-aware LLM	1.48
LLM + MSWSD	2.00
Full DAAF	2.00

Appendix

Table 13: Online environment executions per natural failure. Lower is better.

Failure Attribution for LLM Agents

See all 34 papers

DeFA: Dependency-Guided Failure Attribution for LLM Agents
Oct 1, 2026Bo Deng, Xinlei Zheng, Yi Wei +6Agent Failure Analysis LLM Agent Evaluation
Qwen DianJin Team, Alibaba Cloud Computing · Beihang University
Cascade Failure Attribution for Multi-Agent Systems
Aug 30, 2026Jiayi Zhang, Zexin Wang, Degang Sun +4Agent Failure Analysis Multi-Agent LLM Systems
Computer Network Information Center, Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences
DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems
Sep 4, 2026Zehao Wang, Lanjun Wang, Shilong Jin +2Agent Failure Analysis Multi-Agent LLM Systems
College of Intelligence and Computing, Tianjin University, Tianjin, China · School of New Media and Communication, Tianjin University, Tianjin, China · College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China +1

Oct 1, 2026J/K move · Enter open · S save

DeFA: Dependency-Guided Failure Attribution for LLM Agents

Bo Deng, Xinlei Zheng, Yi Wei, Kang Zhou+5

Qwen DianJin Team, Alibaba Cloud Computing · Beihang University

Agent Failure Analysis LLM Agent Evaluation Failure Attribution for LLM Agents

Open paper AskPDF arXiv

Errors in LLM agent executions and their visible consequences can be separated by many steps, making decisive-error localization a matter of understanding both step content and step dependencies. We introduce DeFA, a dependency-guided framework for agent failure attribution. DeFA first combines protocol relations and semantic dependencies into an event dependency graph spanning the trajectory. It then identifies events that may violate task requirements and traces their sources and subsequent effects to construct a failure propagation graph. Finally, DeFA uses step evidence and the steps' roles in failure propagation to identify the decisive error, responsible agent, and error category. To support long trajectories, DeFA partitions executions into segments and combines the current segment's detailed content with summaries of the other segments, giving local diagnosis access to global execution context. Across Who and When and the Who and When Pro text subset, DeFA achieves the highest responsible-agent and exact step accuracy with all evaluated backbones, and the highest failure-mode accuracy among taxonomy-aligned methods on Pro. Further experiments on image and video trajectories demonstrate its applicability to multimodal failure attribution. Ablations support the contributions of segmentation, the event dependency graph, and the failure propagation graph. Using DeFA's diagnostic feedback for skill evolution in Trace2Skill improves downstream task accuracy by 6-15 percentage points over the native pipeline, showing that the diagnoses can also support agent improvement on subsequent tasks.

Abstract

Failure Attribution for LLM Agents

Explore similar work

DeFA: Dependency-Guided Failure Attribution for LLM Agents

Cascade Failure Attribution for Multi-Agent Systems

DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems