More Agents, More Reliable? System-Level Reliability of Multi-Agent LLMs under Operational-Profile Uncertainty
Organizations: University of Hull, Hull, HU6 7RX, UK · Bilkent University, Ankara, 06800, Türkiye · University of Leeds, Leeds, LS2 9JT, UK
Abstract
Teams increasingly build software from cooperating large language model (LLM) agents, often assuming more agents mean more reliability. Yet the usual evidence, average benchmark accuracy, ignores the input mix a deployed system meets and the rule merging the agents' answers. Building on HIP-LLM, a hierarchical imprecise-probability assessment of single LLMs, we propose a framework combining it with a Bayesian network of the system's failure logic for multi-agent systems. Agent failure-probability intervals propagate through this network, where the combining rule can itself fail, into a reliability envelope showing which evidence limits it. In a pipeline extracting regulated chemical-hazard statements, majority voting over three models protected against the weakest but did not clearly exceed the best, and a voter failing on 5% of demands would largely erase its advantage over the middle model. More agents yield a more defensible reliability claim, not automatically a more reliable system.
Figures & tables
| (a) Agent-level and system-level results | ||||||
|---|---|---|---|---|---|---|
| Configuration | in | in | Envelope in | Envelope in | ||
| GPT-5-mini (best) | 0/20 | 0/70 | ||||
| Gemini-2.5-Flash (middle) | 0/18 | 7/69 | ||||
| Qwen2.5-VL-3B (worst) | 0/20 | 30/70 | ||||
| TMR, propagated | – | – | ||||
| TMR, measured | 0/20 | 1/70 | – | – | ||
Multi-Agent LLM Systems
Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.
See all 1,119 papers