Cooperative multi-agent methods for embodied AI are almost universally evaluated under idealized communication: zero latency, no packet loss, and unlimited bandwidth. Real-world deployment on robots with wireless links, autonomous vehicles on congested networks, or drone swarms in contested spectrum offers no such guarantees. We introduce AgentComm-Bench, a benchmark suite and evaluation protocol that systematically stress-tests cooperative embodied AI under six communication impairment dimensions: latency, packet loss, bandwidth collapse, asynchronous updates, stale memory, and conflicting sensor evidence. AgentComm-Bench spans three task families: cooperative perception, multi-agent waypoint navigation, and cooperative zone search, and evaluates five communication strategies, including a lightweight method we propose based on redundant message coding with staleness-aware fusion. Our experiments reveal that communication-dependent tasks degrade catastrophically: stale memory and bandwidth collapse cause over 96% performance drops in navigation, while content corruption (stale or conflicting data) reduces perception F1 by over 85%. Vulnerability depends on the interaction between impairment type and task design; perception fusion is robust to packet loss but amplifies corrupted data. Redundant message coding more than doubles navigation performance under 80% packet loss. We release AgentComm-Bench as a practical evaluation protocol and recommend that cooperative embodied AI work report performance under multiple impairment conditions.
Figures & tables
Figure 1: Overview of AgentComm-Bench . The benchmark evaluates cooperative methods across three task families under six communication impairment dimensions, producing robustness profiles via standardized metrics.
Figure 2: Communication corruption pipeline. Each message passes through the configurable impairment channel before reaching the receiver. The six dimensions are applied independently by default.
Figure 3: ResilientComm architecture. Redundant message coding sends two copies; staleness-aware fusion weights received messages by estimated age. When both copies are lost, the agent falls back to its most recent received state.
Method
CP (F1)
NAV (WP%)
Search (Rec.%)
No-Comm
95.8
3.6
20.7
Event-Triggered
95.8
96.7
26.8
Compressed
95.8
96.7
26.8
ResilientComm (Ours)
95.8
96.7
26.8
Full-Comm
95.8
96.7
26.8
Table 1: Clean-condition performance ( σ=0 ). Communication provides massive benefit for navigation (waypoint coordination) and moderate benefit for search.
Figure 4: Robustness curves across all task–impairment combinations. Each subplot shows performance (mean ± std over 30 episodes) as communication impairment severity increases. Navigation shows the most dramatic degradation, with all six impairments causing monotonic performance drops from near-perfect coordination to near-random-walk levels. Shaded regions indicate ±1 standard deviation.
Method
CP
NAV
Search
Avg ↓
No-Comm
0.0
0.0
0.0
0.0
ResilientComm (Ours)
28.5
63.6
31.4
41.1
Event-Triggered
28.5
68.6
31.1
42.7
Compressed
28.5
68.4
31.5
42.8
Full-Comm
28.5
68.4
31.5
42.8
Table 2: Mean Normalized Performance Drop (%) across all impairments at maximum severity. Lower is more robust. No-Comm is trivially robust (0% NPD) because it has no communication to degrade.
Figure 5: Normalized Performance Drop (%) at maximum impairment severity. Each cell shows the percentage of clean performance lost. Navigation dominates: stale memory and bandwidth collapse cause > 96% NPD, effectively reverting coordinated agents to random walks. CP shows extreme sensitivity only to content corruption (stale/conflict > 85%).
Figure 6: Method rankings across conditions. Rank 1 is best. Under clean communication, all communicating methods tie. Under impairments, ResilientComm shows rank-1 advantage for NAV packet loss and async; No-Comm dominates CP under content corruption.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Number of agents
4
Episodes per condition
30
Grid size (all tasks)
20 × 20
Message dimension
400 (flattened grid)
Impairment levels per sweep
11
Random seeds
42–71 (per-episode)
Appendix
Table 3: Full benchmark configuration.
Method
Msgs/step
Bits/step
Relative
No-Comm
0
0
0 ×
Full-Comm
12
153,600
1 ×
Compressed
12
19,200
0.125 ×
Event-Triggered
7–10
89,600–128,000
0.6–0.8 ×
ResilientComm
24
307,200
2 ×
Appendix
Table 4: Communication load per step (4 agents, 20 × 20 grid).
Channel
Method
0%
40%
80%
Bernoulli
Full-Comm
96.7
55.0
10.0
( b=0 )
ResilientComm
96.7
86.1
21.9
Bursty
Full-Comm
96.7
58.3
8.3
( b=0.9 )
ResilientComm
96.7
68.3
17.5
No-Comm (constant)
3.6
Appendix
Table 5: NAV waypoint completion (%) under Bernoulli vs. Gilbert-Elliott bursty channel (30 episodes). ResilientComm ’s redundancy advantage is dramatic: at 40% loss, ResilientComm retains 86.1% vs. Full-Comm’s 55.0% under Bernoulli.
Budget
Method
dim
0%
40%
80%
1×
Full-Comm
400
96.7
55.0
10.0
Compressed
400
96.7
55.0
10.0
ResilientComm
200
34.2
31.9
11.4
2×
Full-Comm
400
96.7
55.0
10.0
Compressed
400
96.7
55.0
10.0
ResilientComm
400
96.7
86.1
21.9
Appendix
Table 6: NAV waypoint completion (%) under bandwidth constraints. At 1× budget, ResilientComm ’s 200-dim truncation loses waypoints indexed > 200, severely hurting performance. At 2× budget, ResilientComm dominates via redundancy.
λ
stale=0
stale=4
stale=8
stale=12
stale=20
0.0
95.8
14.0
14.0
14.0
14.0
0.3
95.8
14.0
14.0
14.0
14.0
1.0
95.8
14.0
14.0
14.0
14.0
2.0
95.8
14.0
14.0
14.0
14.0
Appendix
Table 7: Staleness decay ablation: ResilientComm CP F1 (%) across λ values under stale memory (30 episodes). The fusion mechanism dominates; λ has no effect.
Method
CP
NAV
Search
Avg ↑
No-Comm
100.0
100.0
100.0
100.0
ResilientComm (Ours)
73.0
69.8
84.9
75.9
Event-Triggered
73.0
63.4
84.3
73.5
Full-Comm
73.0
63.3
83.0
73.1
Compressed
73.0
63.3
83.0
73.1
Appendix
Table 8: AURC (% of max) averaged across 6 impairment dimensions. No-Comm achieves 100% trivially (flat curves). ResilientComm shows highest AURC among communicating methods due to redundancy.
Figure 7: Sensitivity analysis. Radar plots showing NPD (%) for each method across impairment dimensions. The dominant vulnerability for all communicating methods is to stale memory and conflicting evidence, which corrupt the information being shared.
Multi-agent communication studies how autonomous agents exchange information to coordinate decisions, share knowledge, and act together under constraints.
Agent systems powered by multimodal large language models (MLLMs) have advanced rapidly in recent years, yet existing embodied-agent benchmarks still lack fine-grained diagnostics for multi-agent coordination. Most benchmarks either focus on single-agent task completion or summarize multi-agent behavior with overall task success rates, which can obscure coordination failures such as duplicated work, violations of ordering constraints, resource contention, and desynchronized handoffs. In this paper, we introduce CoCoBench, a construct-level benchmark for evaluating multi-agent embodied coordination in executable household tasks. CoCoBench contains 897 oracle-validated instances organized around four recurring coordination constructs: task allocation, sequential ordering, mutual exclusion, and handoff coordination. In addition to task success rate, CoCoBench provides construct-level scores that measure whether agents coordinate effectively. We evaluate 11 leading MLLMs across different coordination modes, observation inputs, and numbers of agents. The results show that coordination ability is highly construct-specific: strong overall performance does not imply balanced competence across different coordination types. These findings point to new directions for designing targeted model architectures and improving multi-agent coordination ability.
We are entering an era in which individuals and organizations increasingly deploy dedicated AI agents that interact and collaborate with other agents. However, the dynamics of multi-agent collaboration under privacy constraints remain poorly understood. In this work, we present PAC-Bench, a benchmark for systematic evaluation of multi-agent collaboration under privacy constraints. Experiments on PAC-Bench show that privacy constraints substantially degrade collaboration performance and make outcomes depend more on the initiating agent than the partner. Further analysis reveals that this degradation is driven by recurring coordination breakdowns, including early-stage privacy violations, overly conservative abstraction, and privacy-induced hallucinations. Together, our findings identify privacy-aware multi-agent collaboration as a distinct and unresolved challenge that requires new coordination mechanisms beyond existing agent capabilities.
Minjun Park, Donghyun Kim, Hyeonjong Ju +5
Department of Artificial Intelligence, Yonsei University
Existing multi-agent benchmarks primarily test in competitive settings, short-horizon interactions under 20 steps, or simply aggregate individual performance, failing to isolate and highlight genuine collaboration capabilities of LLM-based agents. We introduce AgentWorld, a benchmark of 100 human-annotated tasks (with 100 augmented variants) for evaluating long-horizon, multi-agent collaboration. Tasks span 50+ interaction rounds across a rich MMORPG sandbox and require 3-20 agents with asymmetric roles and abilities to coordinate through communication, joint planning, and resource sharing under a blackbox setting where each agent acts independently without access to others' internal states. To quantify collaboration effectiveness in addition to conventional binary task success, we propose Causal Collaboration Effectiveness (CCE), a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome. Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds. AgentWorld is fully open-source.
Raphael Shu, Yusen Zhang, Young Min Cho +7
OpenAgents · Columbia University · University of Pennsylvania +2