Know the Shape, Find the Fault: Topology-Conditioned Diagnosis of Multi-Agent LLM Failures

Xinwen Liu; Zhuocheng Pan; Isabella Zhu; Jawei Zhang; Xudong Liu; Tianyu Wo

cs.MAOct 7, 2026

Know the Shape, Find the Fault: Topology-Conditioned Diagnosis of Multi-Agent LLM Failures

Authors: Xinwen Liu, Zhuocheng Pan, Isabella Zhu, Jawei Zhang, Xudong Liu, Tianyu Wo

Organizations: Beihang University Beijing, China

Abstract

Multi-agent LLM systems coordinate task execution through exchanges of information among agents. When coordination breaks down, similar symptoms in execution traces can reflect different problems in how information is passed, used, or verified. Communication topology captures how agents exchange information and provides structural cues for distinguishing coordination failure modes. Using these cues for diagnosis requires establishing how topology relates to failure patterns and recovering the relevant structure from execution traces that lack explicit topology labels. We analyze the relationship between communication topology and failure patterns and introduce MAScope, a two-stage framework for topology-conditioned diagnosis. Its Trace Structural Extractor TSE recovers communication topology from heterogeneous execution traces by grounding an interaction graph in message evidence. The Topology-Conditioned Judge TC-Judge then classifies failures using the trace, predicted topology, an empirical failure prior estimated from separate labeled traces, and a short description of topology-specific failure patterns. Under a fixed orchestration structure, the recovered topology can be reused across executions. Experimental results show a statistically significant association between communication topology and failure type, with $χ^2 = 409.9$ and $p = 1.2 \times 10^{-70}$ . On the \num{851} MAST-clean traces, ground-truth topology context raises gpt-mini's Macro-F1 from $0.173$ to $0.350$ . With predicted topology, the pipeline achieves $0.346$ , approaching the trace-only gpt-5.4 baseline of $0.372$ . For \num{1000} traces under a fixed orchestration structure, the projected pipeline cost, including one topology extraction, is approximately $6\%$ of repeated gpt-5.4 diagnosis cost. These results show that topology-conditioned context improves failure diagnosis and supports lower-cost deployment.

Figures & tables

Figure 1. Shared message pool context guides checks of feedback use and task verification. In this trace, the reviewer’s feedback was not acted upon and the requested verification was not performed. A shared-message-pool execution in which a reviewer requests interrupted-transfer handling and verification, but the tester instead checks an unrelated square function and the next review accepts it. The topology view highlights the unaddressed feedback and missing task verification.

Corpus	# traces	Lay.	Cen.	Dec.	SMP
MAST-clean	$851$	$256$	$165$	$0$	$430$
LG-Ctrl	$408$	$102$	$102$	$102$	$102$
Inject-Mix	$961$	$170$	$284$	$507$	$0$

Table 1. MAScope-Bench composition. Lay., Cen., Dec., and SMP denote Layered, Centralized, Decentralized, and Shared Message Pool, based on each declared orchestration graph.

Figure 2. Topology–failure association on MAST-clean ( $851$ traces). Cells show standardised Pearson residuals for the $4\times 14$ contingency table, with $|r|\!\geq\!2$ annotated. The independence test gives $\chi^{2}=409.9$ , $\mathrm{dof}=39$ , and $p<10^{-70}$ . A four-by-fourteen heat map of standardized Pearson residuals between communication topology and MAST failure codes. Each topology row has a different cluster of positive residuals, with Layered associated with information loss, Centralized with verification misalignment, Decentralized with coordination stall, and Shared Message Pool with semantic drift.

Figure 3. MAScope pipeline. TSE converts a heterogeneous trace into an evidence-grounded relation graph and topology label

\hat{\tau}

. TC-Judge combines the trace, predicted topology, frozen topology prior, and failure signature to output a MAST-14 diagnosis. A two-stage pipeline. The first stage adapts and compacts a heterogeneous trace, extracts and validates evidence-grounded relations, and assigns a topology label. The second stage combines the original trace with the topology label, a topology-specific prior, and a failure signature to produce a fourteen-bit MAST diagnosis.

Family	Method	Setting	Macro-F1	Gain over matched $M_{0}$
External protocol	MAST-Judge	gpt-mini	0.131	–
External protocol	MAST-Judge	gpt-5.4	0.363	–
Matched control	$M_{0}$	gpt-mini	0.173	Reference
Matched control	$M_{0}$	gpt-5.4	0.372	Reference
Ours	$M_{4}$	gpt-mini with true topology	0.350	$+0.177$ and $+102\%$
Ours	$M_{4}$	gpt-5.4 with true topology	0.399	$+0.027$ and $+7.3\%$

Table 2. Zero-shot diagnostic results on MAST-clean . No method uses its failure labels for fitting. The final column reports the gain from

M_{0}

M_{4}

on the same model.

Baseline	Construction and comparison role
B1 Opus-Direct	Complete trace, zero-shot model-capacity reference
B2 $M_{0}$	Trace with MAST-14 schema, zero-shot matched control
B3 MAST-Judge ( Cemri et al., 2025 )	Trace with MAST-14 schema, original MAST judge prompt
B4 Flat-GBM ( Ke et al., 2017 )	Seventeen topology and shallow trace features, five-fold learner
B5 Neural-Trace ( Xiao et al., 2024 )	Frozen BGE trace embedding, five-fold textual learner

Table 3. Baselines used in the main and cross-framework comparisons. All inputs are constructed from the same raw traces.

Figure 4. Controlled diagnostic gains from

M_{0}

M_{4}

on MAST-clean . Each comparison fixes the traces, prompt scaffolding, output space, and model. Both models improve, with gpt-mini gaining

102\%

. A grouped bar chart comparing MAST-Judge, the topology-free M-zero judge, and the topology-conditioned M-four judge on gpt-mini and gpt-5.4. M-four raises Macro-F1 from 0.173 to 0.350 for gpt-mini and from 0.372 to 0.399 for gpt-5.4.

Evaluation slice	$N$	C, W, A	Accuracy
MAST-clean by framework
ChatDev	256	211, 44, 1	82.4%
Magentic	165	160, 5, 0	97.0%
MetaGPT	430	427, 0, 3	99.3%
MAST-clean overall	851	798, 49, 4	93.8%
LG-Ctrl by construction topology

Table 4. Topology recovery by framework and controlled topology. C, W, and A denote correct, wrong, and abstained predictions.

Figure 5. Cross-framework scores on MAST-clean . Flat-GBM and Neural-Trace use strict LOFO retraining.

M_{0}

has no fitted component.

M_{4}^{*}

uses a full-corpus topology prior and is a descriptive framework slice rather than a strict LOFO result. Right labels report retention. A dumbbell chart comparing in-distribution and leave-one-framework-out Macro-F1. Flat-GBM falls from 0.490 to 0.226 and Neural-Trace from 0.296 to 0.127, while descriptive zero-shot slices for M-zero and M-four retain 63 percent and 82 percent respectively.

Figure 6. Cost comparison in USD. The upper panel reports measured gpt-5.4 cost and the normalized-token gpt-mini estimate on

851

traces without TSE . The lower panel adds one TSE call and shows deployment cost when the extracted topology is reused. Two cost charts. The upper bar chart compares 8.73 dollars for gpt-5.4 raw-trace diagnosis with 0.52 dollars for gpt-mini using topology on 851 traces. The lower log-scale line chart shows that reusing one extracted topology costs 0.63 dollars for one thousand gpt-mini diagnoses versus 10.26 dollars for gpt-5.4.

Explore similar work

Sep 15, 2026cs.MA

ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failures

LLM-based multi-agent systems can fail even when communication succeeds because agents do not correctly track their peers' roles, knowledge, or intentions. We investigate whether such inter-agent misalignment cases, labelled FC2 in MAST-Data, can be converted into functional partner-state reasoning items. ToMAS applies four explicit convertibility criteria to diagnosed execution traces. A full conversion pass over 242 eligible non-AG2 training traces produced 39 CLEAN items. In an 18-trace reliability pilot, two annotators achieved 94.4% raw agreement and Cohen's kappa = 0.92. We then used the converted items as binary rewards in a small-scale GRPO feasibility experiment with Qwen2.5-1.5B. On a 28-item held-out Magentic GAIA diagnostic, every evaluated condition exceeded the ROUGE-L threshold on the same 2 of 28 items. Post-hoc adapter checks show why: under the learning rate used, the LoRA update remained numerically negligible (max abs Delta W about 7e-6), so all conditions decode identically to the untrained checkpoint. The experiment therefore does not show a training effect and cannot establish one; it reports an executable pipeline together with two limitations that any conclusive study must address: a provenance gap between the training and evaluation items, and lexical-overlap scoring. ToMAS provides a preliminary rubric and pipeline for converting diagnosed coordination failures into trainable partner-state reasoning items and identifies the requirements for a conclusive matched-domain evaluation.

Muhammad Ashar Ishfaq, Glaucia Melo

The Islamia University of Bahawalpur, Pakistan · Toronto Metropolitan University, Toronto, Canada

Multi-Agent Large Language Model Systems Large Language Models Fail Latent Failure Patterns Reasoning Benchmark Pilot Tokan

Ask PDF

May 12, 2026cs.MA

Predictive Maps of Multi-Agent Reasoning: A Successor-Representation Spectrum for LLM Communication Topologies

Practitioners deploying multi-agent large language model (LLM) systems must currently choose between communication topologies such as chain, star, mesh, and richer variants without any pre-inference diagnostic for which topology will amplify drift, converge to consensus, or remain robust under perturbation. Existing evaluation answers these questions only post hoc and only for the task measured. We introduce a structural diagnostic for multi-agent LLM communication graphs based on the successor representation

M = (I - γP)^{-1}

of the row-stochastic communication operator, and we connect three of its spectral quantities, the spectral radius

ρ(M)

, the spectral gap

Δ(M)

, and the condition number

κ(M)

, to three distinct failure modes. We derive closed-form spectra for the chain, star, and mesh under row-stochastic normalization, and validate the predictions on a 12-step structured state-tracking task with Qwen2.5-7B-Instruct over 100 independent trials. The condition number is a perfect rank-order predictor of empirical perturbation robustness (

r_s = 1.0

); the spectral gap partially predicts consensus dynamics (

r_s = 0.5

); and the spectral radius is perfectly \emph{inverted} with respect to cumulative error (

r_s = -1.0

). We trace this inversion to a regime in which linear spectra are blind to non-contracting bias drift, and we propose an affine-noise extension of the predictive map that recovers the empirical ordering. We read this as a first step toward representational, drift-aware structural diagnostics for multi-agent LLM systems, sitting alongside classical spectral and consensus theory.

Ethan Parks, Dalal Alharthi

University of Arizona

Multi-Agent Large Language Model Systems Multi-Agent Reasoning Communication Topologies Successor Representations

Ask PDF

Jul 8, 2026cs.CR

Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

Large language model (LLM) based multi-agent systems enable complex problem solving through coordinated reasoning and action, but their distributed structure also introduces new challenges in diagnosing system-level failures. When an execution fails, identifying which agent is responsible and at what point the trajectory first becomes irreversibly misdirected is difficult due to long-horizon interactions and tightly coupled agent behaviors. In this paper, we study the problem of failure localization in LLM-based multi-agent systems and present AgentLocate, a framework that attributes failures to both a specific agent and the earliest decisive step. AgentLocate combines an LLM-based judging mechanism with multi-perspective verification by independent evaluators, whose assessments are aggregated using a confidence-aware strategy. The resulting feedback is further used to adapt the judge through lightweight fine-tuning, improving attribution quality. We evaluate AgentLocate on two complementary benchmarks covering diverse tasks, agent configurations, and trajectory lengths. Experimental results show that AgentLocate consistently outperforms existing failure localization methods in identifying both responsible agents and failure steps, while remaining efficient in terms of token usage and running time.

Yufei Xia, Anjun Gao, Yueyang Quan +2

University of Louisville · University of North Texas

Model-Based Multi-Agent Systems

Ask PDF