On the Reliability Limits of LLM-Based Multi-Agent Planning

Ruicheng Ao; Siyang Gao; David Simchi-Levi

cs.MAMar 27, 2026

On the Reliability Limits of LLM-Based Multi-Agent Planning

Authors: Ruicheng Ao, Siyang Gao, David Simchi-Levi

Organizations: Operations Research Center, Massachusetts Institute of Technology, Cambridge, MA 02139. · Department of Systems Engineering, City University of Hong Kong, Kowloon, Hong Kong. · Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, MA 02139. · Department of Civil and Environmental Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139.

Abstract

This technical note studies the reliability limits of LLM-based multi-agent planning as a delegated decision problem. We model the LLM-based multi-agent architecture as a finite acyclic decision network in which multiple stages process shared model-context information, communicate through language interfaces with limited capacity, and may invoke human review. We show that, without new exogenous signals, any delegated network is decision-theoretically dominated by a centralized Bayes decision maker with access to the same information. In the common-evidence regime, this implies that optimizing over multi-agent directed acyclic graphs under a finite communication budget can be recast as choosing a budget-constrained stochastic experiment on the shared signal. We also characterize the loss induced by communication and information compression. Under proper scoring rules, the gap between the centralized Bayes value and the value after communication admits an expected posterior divergence representation, which reduces to conditional mutual information under logarithmic loss and to expected squared posterior error under the Brier score. These results characterize the fundamental reliability limits of delegated LLM planning. Experiments with LLMs on a controlled problem set further demonstrate these characterizations.

Figures & tables

LLM-planning element	Model primitive	Decision-theoretic interpretation
Planner / worker / critic / reviewer	node $v$ with decision rule $\pi_{v}$	local decision maker in a delegated network
Shared base model / overlapping retrieved context / common prompt	common signal $B$	shared exogenous information source
Tool call / database query / API output / environment observation	private exogenous signal $Z_{v}$	additional information channel (signal expansion)
Inter-agent prompt relay / summary / scratchpad update	message $T_{u\to v}$	endogenous communication; possible garbling
Terminal transcript / shared workspace	$M$	terminal communication state delivered to the output stage
Verifier test / executable check / external validator	verifier signal $W$	nonredundant inspection stage

Table 1: Mapping LLM-based multi-agent planning to the decision model.

Condition	Model	$N$	Acc.	CI
A ′ (centralized)	4.1-mini	5973	.907	.007
A (centralized)	o4-mini	6000	.899	.008
B 2 (2-agent relay)	4.1-mini	2000	.412	.022
B (3-agent relay)	4.1-mini	5138	.435	.014
B 5 (5-agent relay)	4.1-mini	1993	.225	.018
B (3-agent relay)	o4-mini	1275	.370	.027

Table 2: Empirical results on 200 MMLU questions (conditions A-D) and 200 synthetic KB questions (condition S).

Panel of Figure 1 — Figure 1: Accuracy versus relay length on MMLU (200 questions). A single agent achieves ${\sim}90\%$ on both models. Each additional communication stage degrades accuracy. At five agents, performance falls below the $25\%$ chance baseline.

Panel of Figure 2 — Figure 2: Per-question posterior distortion versus accuracy drop relative to the single-agent baseline. Each point is one MMLU question (averaged across runs). Questions with larger KL divergence suffer larger accuracy losses.

Panel of Figure 3 — Figure 3: Accuracy versus communication stage for posterior vector relay and natural-language prose relay (50 MMLU questions, $n{=}750$ per condition).The posterior interface degrades much more slowly.

Panel of Figure 4 — Figure 4: Redundant versus non-redundant tool access (Corollary 9 ). Both groups use the same single-agent architecture with optional tool lookup. Left (MMLU): the model’s parametric knowledge already covers the questions, so the tool adds little ( $+3.8$ pts). Right (Synthetic KB): 200 fictional entities unknown to the model. Without the tool the model is near chance ( $24.3\%$ ), with the tool accuracy reaches $82.7\%$ ( $+58.4$ pts).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Cond.	Task	Workflow	Signal?	Message	Purpose
(A)	MMLU	Single agent	No	Direct	Centralized baseline
(B 2 )	MMLU	2-stage relay	No	Prose	Short relay
(B)	MMLU	3-stage relay	No	Prose	Delegation w/o new info
(B 5 )	MMLU	5-stage relay	No	Prose	Long relay
(B ${}_{\text{p}}$ )	MMLU	3-stage relay	No	Posterior	Structured interface
(C)	MMLU	Agent + Wikipedia	Yes	Tool	External signal

Appendix

Table 3: Summary of experimental conditions.

Panel of Figure 5 — Figure 5: Information flow for three representative conditions. (A) centralized, where a single agent observes $B$ directly. (B) serial relay, where each stage sees only the previous message. (C) tool-augmented, where the agent acquires an external signal $Z$ . Prompt templates are shown in Boxes A.2 - A.2 .

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 826 papers

Multi-agent decision making: A Blackwell's informativeness approach
May 7, 2026Zheng Zhang, Cuong C. Nguyen, Kevin Wells +1Multi-Agent Debate Question Answering
Centre for Vision, Speech and Signal Processing, University of Surrey
Planning Commitment of LLM under Partial Observability
Oct 15, 2026Cassie Huang, Zhanwei Cao, Yeawon Lee +6LLM Planning LLM Agent Safety
Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds
Jun 28, 2026Mengdie Flora Wang, Haochen Xie, Guanghui Wang +2AI Agent Reliability Multi-Agent LLM Systems
1AWS Generative AI Innovation Center, USA · General Motors (GM), USA

May 7, 2026J/K move · Enter open · S save

Multi-agent decision making: A Blackwell's informativeness approach

Zheng Zhang, Cuong C. Nguyen, Kevin Wells, Gustavo Carneiro

Centre for Vision, Speech and Signal Processing, University of Surrey

Multi-Agent Debate Question Answering LLM Decision-Making

Open paper AskPDF arXiv

The rapid development of large language models (LLMs) has motivated research on decision-making in multi-agent systems, where multiple agents collaborate to achieve shared objectives. Existing aggregation approaches, such as voting and debate, are largely ad-hoc and lack formal guarantees regarding the informativeness of the resulting decisions. In this paper, we provide a principled approach to analyse decisions made in the multi-LLM setting using Blackwell's informativeness framework. Within the Blackwell information-structure abstraction, we show that voting and debate induce information structures that are no more informative than the pooled private information of all agents. This result identifies Bayesian pooled posterior maximisation as an information-theoretic upper-bound decision rule under the Blackwell ordering. Motivated by this theoretical analysis, we introduce a practical method for LLM-based question-answering (QA) tasks that estimates each agent's posterior and approximates the pooled posterior using a product-of-posteriors estimator. Extensive experiments on six QA benchmarks demonstrate that our approach outperforms state-of-the-art multi-LLM debate and voting methods.

Abstract

LLM Agent Reliability

Explore similar work

Multi-agent decision making: A Blackwell's informativeness approach

Planning Commitment of LLM under Partial Observability

Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds