Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems

Yuzhe Zhang; Feiran Liu; Yi Shan; Xinyi Huang; Xin Yang; Yueqi Zhu; Xuxin Cheng; Cao Liu

cs.MAMar 1, 2026

Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems

Authors: Yuzhe Zhang, Feiran Liu, Yi Shan, Xinyi Huang, Xin Yang, Yueqi Zhu, Xuxin Cheng, Cao Liu, +3 more

Organizations: Beijing University of Technology, Beijing, China · Zhejiang University, Hangzhou, China · Meituan LongCat Interaction Team · ETH Zürich, Switzerland · Vector Institute for Artificial Intelligence

Abstract

Large language models are increasingly deployed in multi-agent systems to overcome context limitations by distributing information across agents. Yet whether agents can reliably compute with distributed information, rather than merely exchange it, remains an open question. We introduce SILO-BENCH, a role-agnostic benchmark of 30 algorithmic tasks across three communication complexity levels, evaluating 54 configurations over 1,620 experiments. Our experiments expose a fundamental Communication-Reasoning Gap: agents spontaneously form task-appropriate coordination topologies and exchange information actively, yet systematically fail to synthesize distributed state into correct answers. The failure is localized to the reasoning-integration stage where agents often acquire sufficient information but cannot integrate it. This coordination overhead compounds with scale, eventually eliminating parallelization gains entirely. These findings demonstrate that naively scaling agent count cannot circumvent context limitations, and SILO-BENCH provides a foundation for tracking progress toward genuinely collaborative multi-agent systems. The code is available at https://github.com/jwyjohn/acl26-silo-bench .

Figures & tables

Figure 1: Pipeline of Silo-Bench . Global information is partitioned across $N$ agents, each holding only local data. Agents must communicate through the provided protocol to reconstruct global truth. Success requires effective collaboration strategies. This is an example of the III-21 Distributed Sort (Appendix E .)

Figure 2: Three complexity levels in Silo-Bench characterized by their communication patterns. Level I (Aggregation) : A central agent collects data from all peers via a star topology. Level II (Mesh Network) : Agents exchange information with immediate neighbors through pairwise communication. Level III (Global Shuffle) : All agents must communicate with every other agent, requiring full mesh connectivity.

Figure 3: The three communication protocols employed in Silo-Bench .

By Communication Protocol
Dimension	DeepSeek-V3.1				GPT-OSS-120B				Qwen3-Next-80B-A3B
Dimension	SR $\uparrow$	PCS $\uparrow$	Token $\downarrow$	Den.	SR $\uparrow$	PCS $\uparrow$	Token $\downarrow$	Den.	SR $\uparrow$	PCS $\uparrow$	Token $\downarrow$	Den.
BP	40.4	50.7	297.3	0.93	14.4	33.1	148.7	0.78	4.5	12.2	846.9	0.13
P2P	38.9	50.4	363.1	1.52	20.3	42.6	579.8	2.25	10.5	26.5	909.8	0.62
SFS	31.5	41.0	308.5	1.13	16.0	34.7	212.8	0.74	9.5	18.7	864.4	0.21
By Difficulty Level

Table 1: Overall performance summary across all models and configurations. SR = Success Rate (%), PCS = Partial Correctness Score (%), Token = Token Consumption (tokens/round), Den. = Communication Density. Best results per section in bold .

	Level I (Aggregation)			Level II (Mesh)			Level III (Global Shuffle)
Scale $k$	SR( $N$ =1)	SR( $N$ = $k$ )	RCC	SR( $N$ =1)	SR( $N$ = $k$ )	RCC	SR( $N$ =1)	SR( $N$ = $k$ )	RCC
k = 2	96.7	82.0	15.2%	90.0	62.0	31.1%	80.0	41.0	48.8%
k = 5	93.3	65.0	30.3%	70.0	47.0	32.9%	73.3	22.0	70.0%
k = 10	76.7	51.3	33.1%	73.3	22.0	70.0%	60.0	9.0	85.0%
k = 20	63.3	48.0	24.2%	36.7	14.0	61.8%	43.3	7.0	83.8%
k = 50	33.3	18.0	45.9%	30.0	6.0	80.0%	26.7	0.0	100%

Table 2: Single-agent baseline SR (%), multi-agent SR (%), and Relative Coordination Cost (RCC

=1-\text{SR}(N{=}k)/\text{SR}(N{=}1)

) for GPT-OSS-120B across difficulty levels and scales. RCC columns (shaded) quantify the fraction of single-agent performance lost to coordination overhead. Trends are consistent across all three models; full results in Appendix F .

Level	N=2	N=5	N=10	N=20	N=50	N=100
I	85.0	72.0	68.7	65.7	38.1	40.6
II	61.7	55.3	28.3	29.5	17.4	14.3
III	36.2	17.2	10.0	5.7	0.0	0.0
Avg	61.2	48.5	39.9	33.6	19.0	18.1

Table 3: Success Rate (%) by agent count and difficulty level for DeepSeek-V3.1 (averaged across all protocols).

Figure 4: Scaling behavior across agent counts. (a) Success rates decline for all models as team size increases, with sharp drops beyond

N=20

. (b) Token consumption scales roughly linearly with agent count. (c) Communication density decreases at scale, suggesting coordination sparsification.

Figure 5: Success rate by difficulty level.

Figure 6: Success rate by communication protocol.

Figure 7: Communication heatmaps for representative tasks with $N=20$ agents using DeepSeek-V3.1. Left : I-01 (Global Max) shows emergent leader pattern. Center : II-11 (Prefix Sum) exhibits diagonal pattern indicating partial spatial locality discovery. Right : III-21 (Distributed Sort) shows dense all-to-all communication.

Failure Mode	Count	Percent
Success	153	50.8%
Premature Submission	112	37.2%
Consensus Failure	90	29.9%
Computation Error	86	28.6%

Table 4: Failure mode distribution (categories not mutually exclusive).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Protocol	Description	Available Actions
P2P	Directed messaging via agent-addressed mailboxes	send_message , receive_messages , wait , submit_result
BP	Broadcast messaging to all agents simultaneously	broadcast_message , receive_messages , list_agents , wait , submit_result
SFS	Coordination through shared key-value storage	list_files , read_file , write_file , delete_file , wait , submit_result

Appendix

Table 5: Comparison of communication protocols in Silo-Bench .

Component	Min	Mean	Max	Std
Base Prompt ( $T_{\text{base}}$ )	612	748.8	989	87.3
Data Shard ( $T_{\text{data}}$ )	45	312.4	1,856	298.6

Appendix

Table 6: Token consumption of base prompt and data shard per agent (across all configurations).

Model	Min	Mean	Max	Std
DeepSeek-V3.1	124	8,498.7	98,432	12,847
GPT-OSS-120B	89	2,049.3	45,218	5,632
Qwen3-Next-80B-A3B	156	2,299.3	52,847	6,891

Appendix

Table 7: Communication (

T_{\text{comm}}

) token consumption per agent by model (across all configurations).

Model	Mean Util.	95th Pctl.	Max Util.
DeepSeek-V3.1	7.5%	28.4%	76.9%
GPT-OSS-120B	2.4%	8.2%	35.3%
Qwen3-Next-80B-A3B	2.6%	9.1%	41.3%

Appendix

Table 8: Context window utilization (%) for 128K context models.

Level I: Aggregation (Optimal: Star/Tree Topology, $\mathcal{O}(N)$ messages)
ID	Task Name	Reference	Distributed Adaptation
I-01	Global Maximum	LC-414	Array partitioned; local max $\to$ global aggregation
I-02	Word Frequency	LC-2085	Word lists distributed; count target word globally
I-03	Distributed Vote	LC-169	Vote records partitioned; aggregate to find majority
I-04	Any Match	LC-28	String collection split; detect pattern in any shard
I-05	Range Count	LC-327	Count elements in range across shards

Appendix

Table 9: Complete specification of Silo-Bench tasks with LeetCode references and distributed adaptation descriptions.

Task	BP	P2P	SFS	Avg
Level I: Aggregation
I-01 Global Max	100	100	80	93.3
I-02 Word Frequency	100	52	70	73.9
I-03 Distributed Vote	100	100	100	100.0
I-04 Any Match	100	100	99	99.6
I-05 Range Count	83	67	63	71.0

Appendix

Table 10: Success Rate (%) by task and communication protocol for DeepSeek-V3.1. Tasks with

\geq

50% success are highlighted in gray background.

Model	Protocol	L-I	L-II	L-III
DeepSeek-V3.1	BP	69.7	34.5	14.7
	P2P	62.9	39.9	11.5
	SFS	53.5	30.6	9.0
GPT-OSS-120B	BP	19.6	13.9	9.7
	P2P	34.9	18.5	7.5
	SFS	27.7	11.0	9.1

Appendix

Table 11: Success Rate (%) by model, protocol, and difficulty level.

Model	N=2	N=5	N=10	N=20	N=50	N=100
DeepSeek-V3.1	61.2	48.5	39.9	33.6	19.0	18.1
GPT-OSS-120B	34.4	28.0	14.0	13.2	5.2	6.4
Qwen3-Next-80B-A3B	17.2	9.1	8.6	7.4	5.1	1.3

Appendix

Table 12: Success Rate (%) by model and agent count.

Model	Protocol	L-I	L-II	L-III
DeepSeek-V3.1	BP	0.56	0.82	1.46
	P2P	0.76	1.06	2.83
	SFS	0.82	1.08	1.51
GPT-OSS-120B	BP	0.54	0.84	0.94
	P2P	1.86	2.15	2.73
	SFS	0.73	0.54	0.95

Appendix

Table 13: Communication Density by model, protocol, and difficulty level.

Case	Failure Mode	Description	Key Evidence
1	Consensus Failure	In task I-05, agents communicated extensively but failed to converge, submitting three distinct answers: {1176, 1182, 1167}.	97 of 100 agents submitted 1182; Agent-3 submitted 1176; Agent-55 submitted 1167.
2	Computation Error	Agent-10 in task I-05 received complete data from all peers but computed an incorrect range count.	Submitted 619 instead of correct answer 620. Arithmetic error during final aggregation.
3	Consensus Failure	In task I-05 (different instance), 50 agents split between two answers despite communication.	49 agents submitted 619; Agent-27 submitted 631.
4	Premature Submission	Agent-77 in task I-06 submitted before collecting sufficient data.	Submitted after receiving data from only 28 of 100 agents. Answer: 208; Expected: 114.
5	Consensus Failure	100 agents in task I-06 produced 12 distinct answers for XOR checksum task.	Answers ranged from 42 to 238. Majority (86 agents) converged on 146.

Appendix

Table 14: Representative failure cases from DeepSeek-V3.1 experiments.

	Premature	Consensus	Compute
Premature	112	67	45
Consensus	–	90	52
Compute	–	–	86

Appendix

Table 15: Co-occurrence of failure modes. Diagonal: total occurrences; off-diagonal: joint occurrences.

Metric	Success	Failed
Verification Rate	98.7%	95.9%
Strategy Discussion Rate	93.5%	87.2%
Avg. Messages per Agent	31.2	27.4
Avg. Rounds to Completion	8.3	12.7

Appendix

Table 16: Behavioral comparison between successful and failed runs.

Level	Leader Rate	w/ Leader	w/o Leader
I	27.5%	56.8%	62.1%
II	21.8%	23.5%	59.0%
III	23.8%	0.0%	33.3%

Appendix

Table 17: Leader emergence rates and associated success rates by complexity level.

Scaffold	L-I SR	L-II SR	L-III SR
No scaffold (baseline)	62.9	39.9	11.5
+Planning round	64.3	47.2	17.8
+Protocol reminder	65.1	41.3	12.1
+Scratchpad hint	63.7	44.6	14.9

Appendix

Table 18: Prompt scaffold ablation results under DeepSeek-V3.1 + P2P. SR = Success Rate (%).

Explore similar work

CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning
Aug 28, 2026Yang Chen, Ye-Xin Xie, Lirong Che +8Multi-Agent Coordination AI Agent Benchmarks
Nanjing University · AgiBot · Tsinghua University
Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
Jun 6, 2026Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker +7Multi-Agent Coordination LLM Agent Evaluation
University of Edinburgh · University of Oxford · University College London
Benchmarking Emergent Coordination in Large-Scale LLM Populations: An Evaluation Framework on the MoltBook Archive
Mar 3, 2026Brandon Yee, Pairie KohMulti-Agent Coordination LLM Agent Evaluation
Management Lab, Yee Collins Research Group

Aug 28, 2026J/K move · Enter open · S save

CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning

Yang Chen, Ye-Xin Xie, Lirong Che, Danyang Peng+7

Nanjing University · AgiBot · Tsinghua University

Multi-Agent Coordination AI Agent Benchmarks Multi-Agent Planning Multi-Robot Task Allocation

Open paper AskPDF arXiv

Agent systems powered by multimodal large language models (MLLMs) have advanced rapidly in recent years, yet existing embodied-agent benchmarks still lack fine-grained diagnostics for multi-agent coordination. Most benchmarks either focus on single-agent task completion or summarize multi-agent behavior with overall task success rates, which can obscure coordination failures such as duplicated work, violations of ordering constraints, resource contention, and desynchronized handoffs. In this paper, we introduce CoCoBench, a construct-level benchmark for evaluating multi-agent embodied coordination in executable household tasks. CoCoBench contains 897 oracle-validated instances organized around four recurring coordination constructs: task allocation, sequential ordering, mutual exclusion, and handoff coordination. In addition to task success rate, CoCoBench provides construct-level scores that measure whether agents coordinate effectively. We evaluate 11 leading MLLMs across different coordination modes, observation inputs, and numbers of agents. The results show that coordination ability is highly construct-specific: strong overall performance does not imply balanced competence across different coordination types. These findings point to new directions for designing targeted model architectures and improving multi-agent coordination ability.

Abstract

Explore similar work

CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

Benchmarking Emergent Coordination in Large-Scale LLM Populations: An Evaluation Framework on the MoltBook Archive