Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems

Zhongwen Luan; Xiaoyu Zhang; Ming Hu; Yue Yang; Jiongchi Yu; Xiaohong Chen

cs.AIAug 26, 2026

Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems

Authors: Zhongwen Luan, Xiaoyu Zhang, Ming Hu, Yue Yang, Jiongchi Yu, Xiaohong Chen

Organizations: Xi’an University of Architecture and Technology Xi’an, China

Abstract

As large language model (LLM)-based multi-agent systems (MASs) are increasingly applied to long-horizon complex tasks, their reliability has emerged as the core bottleneck hindering their real-world deployment. Existing MAS debugging and repair methods typically rely on rerunning and resampling the entire execution trajectory. However, a fundamental question remains to be answered: do these methods causally repair MAS failures or merely stochastically repair by leveraging the randomness of LLM sampling? To evaluate the effectiveness of MAS repair methods, we introduce SymTrace, a controlled evaluation framework that records the MAS execution trajectory and establishes intervention anchors. During replay, it effectively reconstructs the execution before the anchor using recorded logs and only regenerates the downstream trajectory, thereby enabling the reliable reproduction of MAS failures. We further construct the dataset SymFail, comprising 536 human-annotated failure trajectories with graph-linked locations, categories, and trace evidence. Based on these foundations, we conduct a large-scale empirical study across three mainstream MAS frameworks. Our findings reveal that existing unguided rerun methods are highly unreliable, exhibiting low failure reproduction and repair rates (only 67.97% and 6.90%, respectively). Building upon these findings, we further explore the effectiveness of a symptom-driven intervention method, which successfully repairs 20.15% of the failed cases (a 191.89% improvement to state-of-the-art repair methods). This study aims to provide actionable insights for MAS debugging and repair research, paving the way for the robust deployment of multi-agent systems.

Figures & tables

Figure 1. Diagnosis and execution instability for the same MAS task. (a) In a Magentic-One distance-query failure, the system makes a routing-derived claim despite receiving no routing result, leading to disagreement between the initial and expert diagnoses. (b) Repeated stochastic reruns of the same task produce different failure types, while some exhibit no detected symptom. A comparison of diagnosis disagreement and stochastic rerun outcomes for the same multi-agent-system task.

Figure 2. Overview of the SymTrace workflow. (Snapshot Mode transforms a native MAS execution into a recorded trajectory through Boundary Logging, Trace Construction, and Replay Bundle. Replay Mode consumes the trajectory, reconstructs the target prefix through Result Injection and Boundary Matching, applies the intervention through Live Resume, and specifies the resulting guarantee through Replay Scope. The lower panels provide an illustrative source and replayed execution.) The SymTrace snapshot and replay workflow, from boundary logging and trace construction to validated prefix replay and live intervention.

Category	Repair signal
$C_{1}$	Implicated constraint and conflicting behavior; request a compliant alternative.
$C_{2}$	Repeated history and unfinished objective; request an action that makes progress.
$C_{3}$	Unresolved runtime condition and its evidence; require resolution before continuing.
$C_{4}$	Explicit plan, action, and outcome; request a decision that resolves their inconsistency.

Table 1. Diagnosis-to-repair mapping. (Categories represent trace-localized intervention signals rather than task-level failure verdicts.)

	$\mathrm{rep}_{1}$ (%)				$\mathrm{rep}_{3}$ (%)
Method	AG2	CA	MO	Total	AG2	CA	MO	Total
Rerun	64.13	77.36	62.06	67.97	36.26	52.72	34.81	41.42
Replay	80.70	81.88	79.74	80.78	50.29	53.26	53.59	52.43

Table 2. Same-failure reproduction rates (%) by MAS on the 536 SymFail source failures. (CA and MO denote CrewAI and Magentic-One; Rerun and Replay denote Unguided Full Rerun and SymTrace Replay. Bold marks the higher value in each column.)

	Group	Cases	$\mathrm{pass@3}$ (%)
	Group	Cases	Rerun	Self	Critic
MAS	AG2	171	8.19	4.68	4.09
	CrewAI	184	3.80	2.17	2.72
	Magentic-One	181	8.84	6.08	4.42
Category	C1	125	9.60	4.80	2.40
	C2	69	4.35	2.90	5.80

Table 3. Task-level

\mathrm{pass@3}

results (%) of the three task-level baselines on SymFail . (The first column groups the same 536 failures by MAS and by primary failure category (C1-C4). Both breakdowns share the single overall Total . ‘Rerun’, ‘Self’, and ‘Critic’ denote Unguided Full Rerun, Self-Reflection, and Critic-Agent.)

Method	Att.	Pass rate (%)
Method	Att.	AG2	CA	MO	Total
Last-Node	1	0.58	1.63	1.66	1.31
Random-Node	1	2.34	4.89	3.87	3.73
Critic-Agent	3	4.09	2.72	4.42	3.73
Self-Reflection	3	4.68	2.17	6.08	4.29
Unguided Full Rerun	3	8.19	3.80	8.84	6.90

Table 4. RQ3 repair utility on the 536 SymFail source failures. (AG2, CrewAI (CA), and Magentic-One (MO) contribute 171, 184, and 181 failures, respectively. ‘Att.’ is the attempt budget: node-level methods use one selective-replay intervention and task-level methods use up to three complete attempts. Bold marks the best result.)

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Initial pattern	Cases	Outcome relative to final annotation
Unanimous	210	all three retained 187; modified 23
2–1 split	214	majority retained 124; minority retained 48; new tuple 42
All distinct	112	one proposed tuple retained 77; new tuple 35

Appendix

Table 5. Relation between the three independent annotations and the final annotation. “Proposed” denotes a complete strict tuple submitted by at least one annotator.

Target	Fleiss’ $\kappa$ (95% CI)	Pairwise	Unanimous (3/3)
Primary category	0.622 (0.545–0.692)	89.86%	459/536 (85.63%)
Exact multi-label category set	0.574 (0.535–0.612)	64.74%	280/536 (52.24%)
C1 presence	0.696 (0.617–0.769)	93.91%	487/536 (90.86%)
C2 presence	0.374 (0.306–0.440)	79.23%	369/536 (68.84%)
C3 presence	0.718 (0.671–0.764)	86.19%	425/536 (79.29%)
C4 presence	0.688 (0.628–0.745)	88.43%	443/536 (82.65%)

Appendix

Table 6. Agreement among the three independent human annotators. Pairwise agreement averages the three annotator pairs; 3/3 agreement is the number of cases in which all annotators make the same decision.

MAS	Method	Method pass@3	Rerun pass@3	$\Delta$ pp [95% CI]	$n_{10}/n_{01}$	Raw $p$	$p_{\mathrm{H}}$
AG2	Self-Reflection	8/171 (4.68%)	14/171 (8.19%)	$-3.51$ [ $-8.19$ , 1.17]	5/11	0.210	0.237
AG2	Critic-Agent	7/171 (4.09%)	14/171 (8.19%)	$-4.09$ [ $-8.77$ , 0.00]	4/11	0.118	0.237
Magentic-One	Self-Reflection	11/181 (6.08%)	16/181 (8.84%)	$-2.76$ [ $-7.73$ , 2.21]	8/13	0.383	0.383
Magentic-One	Critic-Agent	8/181 (4.42%)	16/181 (8.84%)	$-4.42$ [ $-9.39$ , 0.00]	6/14	0.115	0.231
CrewAI	Self-Reflection	4/184 (2.17%)	7/184 (3.80%)	$-1.63$ [ $-4.35$ , 1.09]	2/5	0.453	0.906
CrewAI	Critic-Agent	5/184 (2.72%)	7/184 (3.80%)	$-1.09$ [ $-4.35$ , 1.63]	3/5	0.727	0.906

Appendix

Table 7. Paired RQ2 comparisons against Unguided Full Rerun.

\Delta

is the repair-rate difference between the evaluated method and Unguided Full Rerun.

n_{10}/n_{01}

denotes Method-only/Rerun-only repairs. Each Holm family contains the Self-Reflection and Critic-Agent comparisons within one MAS.

MAS	Comparator	Suspicious-Node	Comparator	$\Delta$ pp [95% CI]	$n_{10}/n_{01}$	Raw $p$	$p_{\mathrm{H}}$
AG2	Unguided Full Rerun	28/171 (16.37%)	14/171 (8.19%)	$+8.19$ [1.75, 14.62]	24/10	0.0243	0.0243
AG2	Self-Reflection	28/171 (16.37%)	8/171 (4.68%)	$+11.70$ [5.26, 18.13]	26/6	$5.35\times 10^{-4}$	$1.07\times 10^{-3}$
AG2	Critic-Agent	28/171 (16.37%)	7/171 (4.09%)	$+12.28$ [7.02, 18.13]	24/3	$4.92\times 10^{-5}$	$1.48\times 10^{-4}$
AG2	Random-Node	28/171 (16.37%)	4/171 (2.34%)	$+14.04$ [8.19, 19.88]	27/3	$8.43\times 10^{-6}$	$3.37\times 10^{-5}$
AG2	Last-Node	28/171 (16.37%)	1/171 (0.58%)	$+15.79$ [10.53, 21.64]	27/0	$1.49\times 10^{-8}$	$7.45\times 10^{-8}$
Magentic-One	Unguided Full Rerun	34/181 (18.78%)	16/181 (8.84%)	$+9.94$ [3.87, 16.02]	27/9	0.00393	0.00393

Appendix

Table 8. Paired RQ3 comparisons.

\Delta

is the repair-rate difference between Suspicious-Node Intervention and the comparator.

n_{10}/n_{01}

denotes Suspicious-only/Comparator-only repairs. Suspicious-Node, Random-Node, and Last-Node use pass@1, whereas the three task-level comparators use pass@3, consistent with the budgets in the main paper. Each Holm family contains the five comparisons within one MAS.

MAS	Method	API calls	Repairs	Repairs/1K calls	Calls/repair
AG2	Last-Node	249	1	4.0	249.0
AG2	Random-Node	293	4	13.7	73.3
AG2	Critic-Agent	295	7	23.7	42.1
AG2	Self-Reflection	314	8	25.5	39.3
AG2	Unguided Full Rerun	400	14	35.0	28.6
AG2	Suspicious-Node	419	28	66.8	15.0

Appendix

Table 9. API-call efficiency on the three MASs evaluated in the main paper. Repairs count native-evaluator-accepted executions. Higher Repairs/1K calls and lower Calls/repair indicate greater repair yield per API call. Bold marks the best value within each MAS.

Figure 3. Suspicious-Node Intervention pipeline. Deterministic rules and a semantic judge jointly evaluate each newly completed node for the C1–C4 symptoms. When the fused suspicion score exceeds the predefined threshold, the ongoing MAS execution is suspended. SymTrace then reconstructs the execution prefix preceding the selected intervention anchor, injects symptom-conditioned repair guidance derived from the triggered rules, and resumes live execution to generate a new downstream trajectory. The suspicious-node intervention pipeline, including rule-based symptom detection, semantic scoring, anchor selection, replay, and live repair.

Category	Operational meaning	Deterministic candidate conditions
C1: Task-constraint violation	The behavior conflicts with an explicit task constraint or output requirement.	The node violates a stated constraint, produces an incompatible output format, contains a placeholder or skipped action, or returns an unresolved request instead of the required result.
C2: Repeated or stalled progress	The history repeats while the objective remains unfinished.	The normalized action fingerprint or tool URL matches a preceding node; the node retries without new parameters or evidence; or another final-style output follows without an action that advances the unfinished objective.
C3: Unresolved runtime condition	The node contains or continues from an unresolved execution, access, or upstream condition.	The node records an error, timeout, or failed tool result; reports inaccessible required evidence; or proceeds toward a final answer while an error remains in the preceding trace.
C4: Plan–action–outcome inconsistency	An explicit plan or claim conflicts with the executed action or observed outcome.	A browser action lacks a URL or uses an invalid view-source: URL; structured data is requested but HTML is returned; completion follows an unsuccessful action; or a tool-dependent task is answered without successful tool evidence.

Appendix

Table 10. Deterministic conditions used to generate trace-localized symptom candidates. The semantic judge may reject a candidate but cannot introduce a category outside this set.

Panel of Table 11 — Table 11. Four stratified gold examples. “Prevalence” is computed from the 536-case finalized ledger by primary category. The displayed examples are qualitative trace evidence, not estimates of model accuracy.

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 826 papers

ARIA: Runtime Inspection and Amendment for Multi-Agent Systems
Oct 14, 2026Bolin Shen, Yushun DongMulti-Agent LLM Systems Multi-Agent Systems
Florida State University
Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Search
Jul 31, 2026Hanxiao Lu, Tianyi ZhangMonte Carlo Tree Search Multi-Agent Systems
Purdue University
MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems
Feb 23, 2026Jin Jia, Zhiling Deng, Zhuangbin Chen +2Multi-Agent LLM Systems LLM Agent Reliability
Sun Yat-sen University Zhuhai, Guangdong, China

Oct 14, 2026J/K move · Enter open · S save

ARIA: Runtime Inspection and Amendment for Multi-Agent Systems

Bolin Shen, Yushun Dong

Florida State University

Multi-Agent LLM Systems Multi-Agent Systems AI Agent Monitoring LLM Agent Training

Open paper AskPDF arXiv

In LLM-based multi-agent systems (MAS), a local mistake can propagate as other agents build on it, turning a correctable error into a failed task. Diagnosing completed runs or repeatedly re-executing them to search for corrections does not provide a learned policy for intervening while execution is still in progress. We present Agent Runtime Inspection and Amendment (ARIA), a framework for learning when and how to repair ongoing multi-agent executions. Our key intuition is to turn execution-verified repairs into reusable supervision, moving repair search into data construction and training so that a learned model can intervene at runtime without repeatedly testing alternatives. First, we collect over 24{,}000 successful and failed trajectories across four multi-agent frameworks, six benchmarks, and eight host models. From these trajectories, we construct the ARIA dataset, to our knowledge the first large-scale dataset pairing failed multi-agent trajectories with execution-verified repairs. It contains 3{,}953 verified repair pairs, each specifying what went wrong, where to intervene, and how execution should continue. Then, we develop the ARIA framework for MAS-agnostic runtime inspection and amendment, designed to integrate with any MAS that exposes execution trajectories and supports continuation replacement. Using verified repairs as training examples, we teach a compact model to recognize when intervention is needed and how to correct an ongoing run. The aim is to learn repair strategies that apply to related failures in new configurations, rather than merely reproduce individual corrections. Finally, the trained ARIA model demonstrates strong repair performance in both post-hoc and runtime settings, with generalization to unseen frameworks and host models. Our code is available at https://github.com/LabRAI/ARIA.

Abstract

LLM Agent Reliability

Explore similar work

ARIA: Runtime Inspection and Amendment for Multi-Agent Systems

Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Search

MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems