Understanding Communication Failures in Multi-Agent LLM Systems

Priya Pitre; Jun Hou; Yi Fang; Naren Ramakrishnan; Xuan Wang

cs.AIOct 23, 2026

Understanding Communication Failures in Multi-Agent LLM Systems

Authors: Priya Pitre, Jun Hou, Yi Fang, Naren Ramakrishnan, Xuan Wang

Organizations: Virginia Tech, Blacksburg, VA 24061, USA

Abstract

Multi-agent LLM systems aim to improve reasoning through debate, evidence sharing, and mutual correction, but interaction can also introduce failures that final accuracy alone cannot explain. Agents may converge too early, ignore dissent, suppress correct minority answers, or preserve a wrong framing across rounds. We present a corpus-grounded taxonomy of communication failures, built from 3,211 failed multi-round trajectories across medical QA, coding, finance, multi-hop reasoning, ethics, and embodied text tasks. Using this taxonomy, we analyze failure distributions across datasets and system designs, round-level error dynamics, recovery hardness, majority transitions, minority-signal suppression, and self-reported confidence. We find that failures are dominated by recovery/correction and consensus/decision problems rather than isolated first-round mistakes, with many emerging after agents first observe peer responses. Based on these findings, we recommend stronger disagreement handling, evidence verification, minority-signal preservation, and recovery checks before finalization.

Figures & tables

Figure 1: Final taxonomy of multi-agent communication failures, organized into parent categories and fine-grained error types.

Figure 2: Our automatic taxonomy creation methodology

Metric	Evaluation level	$n$	Score
Path granularity	Node/path-wise	37	0.946
Sibling coherence	Level-wise	9	0.778
Dimension alignment	Node-wise	43	1.000
Relevance	Node-wise	66	0.712
Coverage	Level-wise	32	1.000

Table 1: Final taxonomy evaluation results. Scores summarize structural quality, coverage, and relevance checks for the final taxonomy across node-, path-, and level-wise evaluations.

Figure 3: Dataset-level failure signatures. QA-style datasets are more consensus-heavy, while coding, finance, and multi-hop settings expose more reasoning, tool-use, and repair-specific failures.

Figure 4: Failure variation across system design choices. Agent count, tool access, orchestration style, and homogeneous composition shift the distribution of failures, but recovery and consensus problems remain common.

	Succ.	Fail.	Ratio
Recovery	0.37	0.94	2.6 $\times$
Consensus	0.51	0.93	1.8 $\times$
Coordination	0.14	0.53	3.6 $\times$
Reasoning	0.35	0.35	1.0 $\times$
Tool / evidence	0.39	0.38	1.0 $\times$
Persistent wrong framing	0.08	0.60	7.5 $\times$

Table 2: Error prevalence (proportion of trajectories) in successful vs. failed debates.

Figure 5: Failure composition across deliberation rounds. Localized error types shift over time: later rounds contain more recovery-oriented failures, while earlier rounds contain more reasoning and consensus-formation errors.

Earlier error	Later error	$P(\text{later}\,\|\,\text{earlier})$	Lift
Premature consensus	Tool result ignored	11.6%	2.79
Premature consensus	Unchallenged evidence	5.4%	1.87
Premature consensus	Correction ignored	16.5%	1.62
Edge-case failure	Failed recalculation	22.5%	1.51
Unsupported assumption	Tool result ignored	11.4%	1.49
Oscillation	Question misread	10.4%	1.39

Table 3: Cross-type temporal error cascades. Each row reports how often a later failure type appears after an earlier failure type first appears in the same trajectory. Lift compares this with how often the later error appears after the same round in debates without the earlier error; values above 1 indicate that the later error is unusually likely after the earlier error.

Figure 6: Hardest mapped suberrors to recover from. Bars show the probability that an error appearing before a debate’s final round is still present in that final round. Group-dynamic and repair-oriented failures are the most persistent.

Dataset	Rec.	Corrupted	Failed Recovery	Preserved	Ind. Rescue	Ind. Supp.
FinQA	12.83	2.00	50.33	34.83	6.17	16.33
HotpotQA	24.92	4.83	33.83	36.42	9.92	12.33
MMLU	13.21	1.42	32.54	52.83	7.00	9.17
MedQA	56.19	0.38	36.61	6.82	0.38	1.77
MuSiQue	16.25	1.75	62.67	19.33	6.42	15.17
Ethics	3.25	12.25	26.25	58.25	1.88	28.75

Table 4: Initial-majority transition dynamics by dataset. Recovered, corrupted, failed recovery, and preserved form the main transition partition. Individual rescue and individual suppression are overlapping diagnostics for cases where initially correct minority agents do or do not change the final answer.

Metric	Ans. change	Split	Round $\geq$ 2	Round +split	LLM judge	Ours
Premature consensus
Prec.	0.23	0.45	0.22	0.45	0.46	0.60
Rec.	0.30	0.97	0.34	0.97	0.46	0.69
F1	0.26	0.61	0.26	0.62	0.46	0.64
Acc.	0.24	0.46	0.17	0.47	0.53	0.66
AUROC	0.39	0.56	0.33	0.70	0.57	0.75

Table 5: Predicting errors one round before they start. After each round, each method predicts whether the error begins in the next round. Precision, recall, F1, and accuracy are computed per debate: a warning counts only if it fires before onset. AUROC is computed per round. Parentheses give affected debates over all debates; correct and incorrect debates are balanced. Leave-one-pair-out evaluation; best value per row in bold.

Error	No trigger	Ours	$\Delta$
Premature consensus	0.56	0.66	+0.10
Failed recovery	0.55	0.58	+0.03
Unknown recovery	0.69	0.89	+0.20

Table 6: Accuracy with and without the trigger. Debate-level accuracy when never warning versus with our trigger.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Provider / Infrastructure	Approx. input toks.	Approx. output toks.	Approx. cost
Multi-agent trajectory generation	GPU cluster / DeepInfra	25–35M	8–12M	$25.00
Trajectory card generation	OpenAI	14–18M	3–5M	$6–10
Candidate tag extraction	OpenAI	4–6M	1.5–2.5M	$3–5
Taxonomy mapping and cleanup	OpenAI	6–10M	1–2M	$3–6
Cluster review and width/depth expansion	OpenAI	1–3M	0.3–0.8M	$1–3
LLM-judge trigger baseline	OpenAI	$<$ 0.1M	$<$ 0.1M	$<$ $1

Appendix

Table 7: Approximate inference cost by pipeline component. DeepInfra cost reflects observed billing. ARC runs used institutional GPU infrastructure and are therefore reported as compute usage rather than direct API cost. OpenAI costs are estimated from approximate input/output token counts and provider-reported per-token prices at the time of experimentation.

Work	Comm. Error?	Automatic?	# Domains	# Traces	Failure Categories / Focus	Tracks Answer Evolution?	Multi-error?
Cemri et al. (2026)	$\triangle$	✗	7 frameworks	1600+	System design; inter-agent misalignment; task verification	✗	✓
Zhang et al. (2025)	✗	$\triangle$	2 datasets	127 systems / 184 annotations	Responsible-agent identification; decisive failure-step attribution	$\triangle$	✗
Zhu et al. (2025)	✗	$\triangle$	3 tasks	NR	Planning; tool use; memory; reflection; action; system-level failures	✗	✓
Bi et al. (2026)	✗	✗	10 systems	307	Root causes in platform-orchestrated agentic workflows; platform/system failures	✗	✓
Barke et al. (2026)	✗	✓	3 domains	115	Critical failure steps; root-cause attribution; repair-oriented diagnosis	✗	✓
Ma et al. (2026)	✗	✗	2 platforms / 10 systems	307	Lifecycle failures; planning; execution; tool use; orchestration failures	✗	✓

Appendix

Table 8: Comparison with prior taxonomy, dataset, and benchmark resources for agent and multi-agent failure analysis. Comm. Error? indicates whether inter-agent communication failures are a primary object of analysis. Automatic? indicates whether taxonomy construction or annotation is automatic/semi-automatic rather than primarily human-authored. Tracks Answer Evolution? indicates whether the work explicitly follows answer changes across agents and rounds. Multi-error? indicates whether multiple failure mechanisms can be assigned to the same trace. ✓= yes, ✗= no,

\triangle

= partial, NR = not reported.

Work	Paper Type	Comm. Relevant?	Taxonomy Resource?	Main Contribution	Distinction from Our Work
Wynn et al. (2025)	Empirical debate study	✓	✗	Shows that multi-agent debate can reduce accuracy through peer influence, agreement pressure, and correctness shifts over rounds.	This work motivates the importance of communication failures, but it does not construct a general taxonomy or annotation framework for such failures. Our work turns this kind of observation into a broader framework that identifies where communication failures originate, how they propagate, whether they are corrected, and how they affect finalization across tasks and orchestration settings.
Bi et al. (2026)	Diagnostic method	✗	✗	Uses behavioral abstractions or diagnostic insights to explain and localize agent failures.	This work focuses on improving diagnosis through higher-level behavioral summaries, but it does not model how agents influence one another over multiple deliberation rounds. Our work explicitly tracks answer movement, correction attempts, evidence uptake, confidence shifts, dissent, and finalization decisions, making the communication process itself the object of analysis.

Appendix

Table 9: Related empirical and diagnostic studies that motivate our work but are not primarily taxonomy, dataset, or benchmark resources for communication-failure analysis.

Setting	Overrepresented suberror	In setting	Elsewhere	Lift
Dataset-specific failures
FinQA	Calculation / representation error	25.9	2.7	9.55 $\times$
LiveCodeAgent	Edge-case handling failure	27.9	3.0	9.27 $\times$
LiveCodeAgent	Failed computational repair	37.1	11.9	3.12 $\times$
MBPP	Failed recovery after disagreement	31.9	18.3	1.74 $\times$
Agent-count-specific failures

Appendix

Table 10: Suberrors overrepresented in specific datasets and system settings. Percentages report how often a suberror appears within the named setting versus outside it. Lift is the ratio between the two rates.

Level	Failure subtype	Count	% of cases
Child suberrors
Child	Anchor-driven recovery failure	1,067	33.2
Child	Social convergence without verification	1,053	32.8
Child	Unstable coordination	856	26.7
Mapped suberrors
Mapped	Persistent wrong framing after correction	1,993	62.1

Appendix

Table 11: Most frequent fine-grained failure subtypes. Counts are computed over 3,211 failed multi-agent trajectories. Labels are multi-label, so percentages do not sum to 100%.

Failure pair	Count	% of all cases
Consensus + Recovery	2,523	78.6
Coordination + Recovery	1,589	49.5
Tool/evidence + Recovery	1,341	41.8
Reasoning + Recovery	1,318	41.0

Appendix

Table 12: Co-occurrence between recovery failures and other broad error types. Counts are computed over 3,211 failed trajectories. The high overlap shows that many failures involve converting disagreement, evidence, or correction signals into a final repaired answer.

Figure 7: Broad failure category distribution. Recovery/correction and consensus/decision failures dominate the failed trajectory set, indicating that many errors are failures of deliberative repair rather than isolated first-pass mistakes.

Figure 8: Localized error counts by round. Error events peak after the first exchange, indicating that many failures emerge as agents react to peer responses rather than only in the initial answer.

#Agents	Recovered	Corrupted	Failed Recovery	Preserved	Ind. Rescue	Ind. Supp.
3	19.50	3.37	41.51	35.62	7.87	11.37
5	22.19	3.78	39.27	34.76	4.11	14.67
7	22.06	4.00	40.13	33.81	2.50	17.70

Appendix

Table 13: Initial-majority transition dynamics by number of agents. Recovery rates remain broadly similar as the number of agents increases, while individual suppression rises with group size.

Figure 9: Model-only taxonomy classification by error label. Each point reports per-label F1 when a model classifies parent or leaf error labels from a whole debate trace. Low and uneven performance, especially on fine-grained labels, motivates a systematic taxonomy construction and annotation pipeline.

Setting	3 Base Llama3.2, Mistral, Llama3.3	3 Alt gpt-oss/Gemma/Qwen	5 Agents Base + gpt-oss, Gemma	7 Agents + Qwen3, gpt-oss-120B
FinQA
Tools	47	51	48	–
Tools + Roles	49	47	44	–
HotpotQA
Question-only	39	32	34	37
Roles	71	67	75	76

Appendix

Table 14: Run-level accuracy by dataset and agent configuration. Values report accuracy (%). Column headers summarize the default agent compositions from the run summary. Role-based rows use task-specific agent roles with the corresponding base, alternate, expanded, or seven-agent role configurations.

Figure 10: Debate round dynamics. Left: percentage of debates terminating at each round. Right: final correctness by debate length. One-round consensus is usually reliable, while multi-round debates account for most failures despite also producing many correct answers.

Figure 11: Taxonomy saturation curve. Parent and child categories saturate quickly, while fine-grained leaf labels require larger samples because several leaf errors are rare.

Quantity	Correct	Incorrect	$p$ -value	Interpretation
Confidence gain from first to final round	+5.54	+10.49	$5.30{\times}10^{-53}$	Wrong traces gain more confidence
Confidence gain after answer flip	+10.56	+7.42	$1.79{\times}10^{-16}$	Flips increase confidence in both groups
Final-round confidence	93.10	89.34	$3.94{\times}10^{-40}$	Correct is higher, but wrong remains high

Appendix

Table 15: Self-reported confidence dynamics. The first two rows report average confidence changes; the final row reports average final-round confidence.

p

-values compare correct and incorrect traces for each quantity.

Dataset	Debates w/ UNKNOWN (%)	Dominant UNKNOWN type	Blank (%)	Reasoning but UNKNOWN (%)	Format fail (%)	Recovery (%)	Persistence (%)	Main pattern
Ethics	$42.8\pm 26.0$	Blank response	43.0	23.5	4.5	52.0	48.0	Mixed; often blank-driven
FinQA	$25.0\pm 16.3$	Blank response	54.2	31.7	2.1	75.0	25.0	UNKNOWN often resolves
HotpotQA	$39.9\pm 29.7$	Blank response	49.8	46.0	1.7	60.7	39.3	Mixed but recoverable
MedQA	$49.8\pm 18.6$	Reasoning without answer	10.2	75.8	9.9	10.7	89.3	Highly persistent
MMLU	$42.7\pm 29.7$	Reasoning without answer	19.6	78.4	1.2	63.6	36.4	Reasoned uncertainty often resolves
MuSiQue	$67.2\pm 32.7$	Reasoning without answer	46.6	48.0	0.8	40.1	60.0	Frequent and often persistent

Appendix

Table 16: UNKNOWN answer behavior by dataset. Blank responses and reasoning-but- UNKNOWN responses account for most UNKNOWN cases, while explicit format failures are relatively rare. Recovery and persistence measure whether an agent that produced UNKNOWN in one round gave a determinate answer or remained UNKNOWN in the next round.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,034 papers

When Messages Go Wrong: Understanding Communication Faults in LLM-Based Multi-Agent Systems
Oct 11, 2026Jinhao Zhou, Xiaoyu Zhang, Jiongchi Yu +2Multi-Agent LLM Systems Multi-Agent Communication
Waseda University, Japan · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Nanyang Technological University, Singapore +2
What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systems
May 19, 2026Yong Jin Chun, Iftekhar AhmedMulti-Agent Collaboration Multi-Agent Systems
Department of Informatics University of California, Irvine Irvine, CA, 92618
Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systems
Oct 1, 2026Shixuan Li, Wei Yang, Peiyu Zhang +3Multi-Agent LLM Systems AI Agent Auditing
Ming Hsieh Department of Electrical and Computer Engineering University of Southern California, Los Angeles, CA 90089, USA · Thomas Lord Department of Computer Science University of Southern California, Los Angeles, CA 90089, USA

Oct 11, 2026J/K move · Enter open · S save

When Messages Go Wrong: Understanding Communication Faults in LLM-Based Multi-Agent Systems

Jinhao Zhou, Xiaoyu Zhang, Jiongchi Yu, Fahao Chen+1

Waseda University, Japan · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Nanyang Technological University, Singapore+2

Multi-Agent LLM Systems Multi-Agent Communication LLM Agent Reliability

Open paper AskPDF arXiv

LLM-based multi-agent systems (LLM-MAS) use specialized agents that exchange messages to accomplish complex tasks. Their reliance on networked protocols, remote tools, and model endpoints exposes communication to message loss, delay, and context expiration. Because agents often exchange open-ended natural language, anomalies can alter task-relevant information while leaving messages processable, allowing semantic errors to propagate through subsequent interactions. Systematic studies of faults in inter-agent communication remain limited. This paper presents a large-scale empirical study of how communication faults manifest, propagate, and affect task outcomes in LLM-MAS. We construct a cross-layer taxonomy linking lower-stack disturbances to application-visible faults and develop a fault-injection and tracing pipeline. Our evaluation covers more than 29,000 executions across 100 tasks, five domains, three workflow topologies, and three model configurations. We find that 20.7% of failures with complete detection records occur without an immediate failure signal, reaching 76.2% for partial-content faults. Flat maintains 53.0% task success under communication faults, compared with 3.3% and 3.2% for Sequential and Hierarchical, respectively. This advantage primarily arises when an alternative information path preserves the evidence required by downstream agents. We also evaluate execution-side and evidence protection mechanisms and find that they recover different failure cases. Evidence recovery depends on whether trustworthy information remains accessible through an unaffected path. These observations provide empirical insights into communication-fault propagation, workflow resilience, and protection design for LLM-MAS.

Abstract

Multi-Agent LLM Systems

Explore similar work

When Messages Go Wrong: Understanding Communication Faults in LLM-Based Multi-Agent Systems

What Do Agents Communicate? Characterizing Information Exchange in Multi-Agent Systems

Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systems