Organizations: Beijing Institute of Technology, Beijing, China · Zhipu AI, Beijing, China · Tsinghua University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China
Agentic systems augment large language models with external tools and iterative decision making, enabling complex tasks such as deep research, function calling, and coding. However, their long and intricate execution traces make failure diagnosis and root cause analysis extremely challenging. Manual inspection does not scale, while directly applying LLMs to raw traces is hindered by input length limits and unreliable reasoning. Focusing solely on final task outcomes further discards critical behavioral information required for accurate issue localization. To address these issues, we propose TraceSIR, a multi-agent framework for structured analysis and reporting of agentic execution traces. TraceSIR coordinates three specialized agents: (1) StructureAgent, which introduces a novel abstraction format, TraceFormat, to compress execution traces while preserving essential behavioral information; (2) InsightAgent, which performs fine-grained diagnosis including issue localization, root cause analysis, and optimization suggestions; (3) ReportAgent, which aggregates insights across task instances and generates comprehensive analysis reports. To evaluate TraceSIR, we construct TraceBench, covering three real-world agentic scenarios, and introduce ReportEval, an evaluation protocol for assessing the quality and usability of analysis reports aligned with industry needs. Experiments show that TraceSIR consistently produces coherent, informative, and actionable reports, significantly outperforming existing approaches across all evaluation dimensions. Our project and video are publicly available at https://github.com/SHU-XUN/TraceSIR.
Figures & tables
Figure 1: Comparison of existing methods and TraceSIR for agentic execution trace analysis and reporting.
Figure 2: The overall architecture of TraceSIR .
Scenario
Method
Backbone
Overall Score
OS
EA
RCA
OA
OI
Ranking
Deep Research
ClaudeCode
GLM-5
55.0
5.5
4.5
5.5
6.0
6.0
4
ClaudeCode
Claude-4.6
66.0
6.5
7.5
5.5
7.0
6.5
2
TraceSIR
GLM-5
65.0
7.5
6.5
5.5
6.0
7.0
3
TraceSIR
Claude-4.6
81.0
8.5
8.5
8.0
7.0
8.5
1
Function Calling
ClaudeCode
GLM-5
40.0
5.0
3.5
4.5
3.0
4.0
4
ClaudeCode
Claude-4.6
74.0
7.5
7.5
8.5
6.5
7.0
2
Table 1: Comparison of analysis report quality under the ReportEval protocol using human evaluation . TraceSIR is our system while ClaudeCode is the baseline.
Scenario
Method
Backbone
Overall Score
OS
EA
RCA
OA
OI
Ranking
Deep Research
ClaudeCode
GLM-5
82.7
8.7
8.0
7.3
9.0
8.3
4
ClaudeCode
Claude-4.6
90.0
9.3
8.7
9.0
9.0
9.0
2
TraceSIR
GLM-5
88.0
9.0
8.0
9.0
9.0
9.0
3
TraceSIR
Claude-4.6
91.3
9.3
9.0
9.3
9.0
9.0
1
Function Calling
ClaudeCode
GLM-5
80.7
8.3
7.3
7.7
9.0
8.0
4
ClaudeCode
Claude-4.6
88.0
8.7
8.3
9.0
9.0
9.0
3
Table 2: Comparison of analysis report quality under the ReportEval protocol using LLM-as-a-judge . TraceSIR is our system while ClaudeCode is the baseline.
Figure 3: The demonstration of TraceSIR.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: An excerpt from a Chinese analysis report generated by TraceSIR using Claude-4.6 on the Deep Research scenario of BrowseComp.
Category
Capability
TraceSIR
ClaudeCode
System Design
1. Multi-agent architecture explicitly designed for analysis and reporting, with separated roles for trace structuring, diagnosis, and reporting
✓
△
2. Fine-grained, per-case diagnostic outputs with structured fields
✓
△
3. Native cross-case aggregation and statistical analysis over all traces
✓
△
4. Incremental analysis with resume support for large trace collections
✓
×
5. Configurable analysis and report constraints via explicit settings
✓
△
Methodology
1. Full execution trace modeling with explicit Thought–Action–Observation
✓
×
Appendix
Table 3: Capability-level comparison between TraceSIR and ClaudeCode. ✓ indicates native and systematic support, △ indicates partial or non-systematic support, and × indicates lack of support.
As multi-agent AI systems are increasingly deployed in real-world settings - from automated customer support to DevOps remediation - failures become harder to diagnose due to cascading effects, hidden dependencies, and long execution traces. We present AgentTrace, a lightweight causal tracing framework for post-hoc failure diagnosis in deployed multi-agent workflows. AgentTrace reconstructs causal graphs from execution logs, traces backward from error manifestations, and ranks candidate root causes using interpretable structural and positional signals - without requiring LLM inference at debugging time. Across a diverse benchmark of multi-agent failure scenarios designed to reflect common deployment patterns, AgentTrace localizes root causes with high accuracy and sub-second latency, significantly outperforming both heuristic and LLM-based baselines. Our results suggest that causal tracing provides a practical foundation for improving the reliability and trustworthiness of agentic systems in the wild.
Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.
LLM-powered agents increasingly tackle complex tasks by invoking tools, querying databases, executing code, and manipulating intermediate artifacts. These agents follow trajectories that are typically stored as chronological logs, obscuring the underlying dataflow -- the dependencies between their actions and the artifacts they create and manipulate. This limits developers' ability to understand the agents' trails, compare executions, debug failures, and re-use the computations. We present AgentTrails, a prototype system for agent provenance and sensemaking. AgentTrails converts raw trajectories into structured provenance graphs, where tool calls are modeled as computational actions and inputs and outputs as data artifacts. The system supports the comparison of executions by placing multiple provenance graphs on a shared canvas and constructing a joined quotient graph that aligns recurring tools, artifacts, and dependency structures across trajectories. On top of this representation, AgentTrails supports pattern extraction, downstream analysis, and skill abstraction. We demonstrate AgentTrails on real-world agent trajectories, showing that it reveals hidden dependencies, aligns divergent executions, and surfaces recurring tool-use patterns beyond chronological logs.