cs.ROAug 28, 2026
SaveCoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning
Organizations: Nanjing University · AgiBot · Tsinghua University
Abstract
Agent systems powered by multimodal large language models (MLLMs) have advanced rapidly in recent years, yet existing embodied-agent benchmarks still lack fine-grained diagnostics for multi-agent coordination. Most benchmarks either focus on single-agent task completion or summarize multi-agent behavior with overall task success rates, which can obscure coordination failures such as duplicated work, violations of ordering constraints, resource contention, and desynchronized handoffs. In this paper, we introduce CoCoBench, a construct-level benchmark for evaluating multi-agent embodied coordination in executable household tasks. CoCoBench contains 897 oracle-validated instances organized around four recurring coordination constructs: task allocation, sequential ordering, mutual exclusion, and handoff coordination. In addition to task success rate, CoCoBench provides construct-level scores that measure whether agents coordinate effectively. We evaluate 11 leading MLLMs across different coordination modes, observation inputs, and numbers of agents. The results show that coordination ability is highly construct-specific: strong overall performance does not imply balanced competence across different coordination types. These findings point to new directions for designing targeted model architectures and improving multi-agent coordination ability.
Figures & tables
Figure 1: Motivation of CoCoBench. Existing benchmarks either focus on single-agent task success or multi-agent final completion, whereas CoCoBench explicitly evaluates coordination ability through construct-level metrics.
Figure 2: Comparison with related embodied benchmarks. Left: coordination-dimension breadth versus task-type diversity, where breadth counts the coverage of T, S, M, and H. Right: benchmark attributes, where Cases denotes the number of evaluation cases; Dims. lists the covered coordination dimensions; Cent. / Dec. , MM , and Role indicate centralized/decentralized modes, multimodal observations, and heterogeneous roles, respectively. TS denotes trajectory-level construct scores. T/S/M/H represent task allocation, sequential ordering, mutual exclusion, and handoff coordination.
Figure 3: Overview of the CoCoBench framework. CoCoBench evaluates multi-agent embodied coordination through a three-stage pipeline: construct-level task design, agent assignment, and skill-based joint planning. The benchmark covers four coordination constructs and maps them to executable household tasks with explicit agent roles and high-level skill interfaces.
Figure 4: Qualitative examples across four coordination dimensions.
| Overall | D1: Task allocation | D2: Sequential ordering | D3: Mutual exclusion | D4: Handoff coordination | ||||||
| Model | SR | CS | SR | CS | SR | CS | SR | CS | SR | CS |
| Proprietary (API) | ||||||||||
| GPT-5.6-sol | 84.8 | 0.90 | 57.3 | 0.63 | 90.2 | 0.96 | 96.5 | 1.00 | 94.2 | 0.99 |
| GPT-5.4-mini | 33.6 | 0.51 | 81.7 | 0.66 | 30.7 | 0.60 | 10.9 | 0.59 | 12.9 | 0.18 |
| Claude Opus 4.8 | 78.5 | 0.87 | 38.1 | 0.50 | 94.7 | 1.00 | 93.9 | 0.99 | 85.7 | 0.99 |
| Claude Haiku 4.5 | 63.1 | 0.71 | 89.4 | 0.48 | 47.1 | 0.71 | 75.7 | 0.93 | 40.6 | 0.70 |
Table 1: Main results on CoCoBench. Results are reported on the full 897-instance set under the centralized image-observation protocol. SR = task success rate (%); CS = construct score (0–1). Best per column in bold .
Figure 5: Construct correlations. Pairwise Pearson correlations between coordination scores on GPT-5.6 trajectories.
Figure 6: Architecture effects on the 240-instance subset. Signed change in SR and CS of the two decentralized policies relative to the centralized baseline, per model and construct, pooled over image and blind observations.
Figure 7: Observation ablation on the 240-instance subset. SR and CS under the image versus blind settings, per model and construct, pooled over the three policy architectures.
Figure 8: Scaling with team size. Per-model SR (a) and CS (b) across two-, three-, and four-agent instances.
Figure 9: Failure modes and legality. Per-model terminal-state taxonomy (a) and success rate versus legal-plan rate (b).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Coordination Type | Task Type | Instruction Example | Description |
|---|---|---|---|
| Store Groceries | Put the Potato and Lettuce in the Fridge, and turn off the LightSwitch. Put the Potato in the Fridge, put the Lettuce in the Fridge, and make sure the LightSwitch is off. | Agents divide independent household subtasks, such as storing food and switching off a light, and finish all objectives without repeated actions. | |
| D1 Task Allocation | Sort & Store | Put the CreditCard on the CoffeeTable, the CellPhone on the Desk, and the Pencil on the ArmChair. | Agents sort multiple personal objects to designated receptacles and split object-target pairs across the team. |
| Tidy Drawers | Put the CreditCard in Drawer_1, the CellPhone in Drawer_2, and the Pencil in Drawer_3. | Agents place personal items into assigned drawers, opening drawers before placement when needed. | |
| Stock Cabinet | Put the Apple and Lettuce in the Cabinet, then close the Cabinet after both items are inside. | Agents load kitchen items into a cabinet and close it only after all required items have been placed. | |
| D2 Sequential Ordering | Load in Order | Put the KeyChain, Watch, CreditCard, and RemoteControl in the Box, then close the Box. | Agents load several objects into a container and perform the final close action after the loading sequence is complete. |
| Fill Drawers in Order | Put the KeyChain, Watch, CreditCard, and RemoteControl in the Drawer, then close the Drawer. | Agents place several personal objects into one drawer and close it only after all placements are complete. |
Table 2: Definitions and instruction examples of ten coordination-oriented task types in CoCoBench.
Figure 10: CoCoBench composition across coordination constructs and task families.
| Model Name | Creator | Full Name / Identifier |
| GPT-5.6-sol | OpenAI | gpt-5.6-sol |
| GPT-5.4-mini | OpenAI | gpt-5.4-mini |
| Claude Opus 4.8 | Anthropic | claude-opus-4.8 |
| Claude Haiku 4.5 | Anthropic | claude-haiku-4.5 |
| Qwen3.6-Plus | Qwen | qwen3.6-plus |
| Kimi-K2.6 | Moonshot AI | kimi-k2.6 |
Table 3: Models used in our evaluation.
| Action Type | Target Object |
|---|---|
| Find | AlarmClock, Apple, ArmChair, BasketBall, Bathtub, Bed, Book, Box, Bread, ButterKnife, Cabinet, Candle, Cart, CellPhone, CD, Chair, Cloth, CoffeeMachine, CoffeeTable, CounterTop, CreditCard, Cup, Desk, DeskLamp, DishSponge, DiningTable, Dresser, Drawer, Egg, Faucet, FloorLamp, Fork, Fridge, GarbageCan, GlassBottle, HandTowel, Kettle, KeyChain, Knife, Ladle, Laptop, Lettuce, LightSwitch, Microwave, Mug, Newspaper, Ottoman, Pan, PepperShaker, Pencil, Pen, Pillow, Plate, Plunger, Potato, RemoteControl, Safe, SaltShaker, Shelf, SideTable, Sink, SoapBar, SoapBottle, Sofa, Spatula, SprayBottle, Statue, StoveBurner, TennisRacket, TissueBox, Tomato, Toilet, ToiletPaper, ToiletPaperHanger, Vase, Watch, WateringCan, WineBottle |
| Explore | forward, back, left, right, turn_left, turn_right, look_up, look_down |
| Wait | None |
| PickUp | AlarmClock, Apple, BaseballBat, BasketBall, Book, Bowl, Box, Bread, ButterKnife, Candle, CD, CellPhone, Cloth, CreditCard, Cup, DishSponge, Egg, Fork, GlassBottle, HandTowel, Kettle, KeyChain, Knife, Ladle, Laptop, Lettuce, Mug, Newspaper, Pan, Pen, Pencil, PepperShaker, Plate, Plunger, Potato, RemoteControl, SaltShaker, SoapBar, SoapBottle, Spatula, SprayBottle, Spoon, Statue, TennisRacket, TissueBox, Tomato, Vase, Watch, WateringCan, WineBottle |
| Put | ArmChair, Bed, Box, Cabinet, Chair, CoffeeTable, CounterTop, Desk, DiningTable, Drawer, Dresser, Fridge, Microwave, Plate, Safe, Shelf, SideTable, Sink, Sofa |
| Drop | Object in hand |
Table 4: Action types and target object categories exposed to the planner.
| Model | Prog. (fail) | Steps (succ.) | Inv. act. | Dep | Occ | Aff | Ill |
| Proprietary (API) | |||||||
| GPT-5.6-sol | 42.0 | 15.8 | 1.8 | 29 | 7 | 2 | 0 |
| GPT-5.4-mini | 45.0 | 16.7 | 0.0 | 95 | 6 | 3 | 0 |
| Claude Opus 4.8 | 39.8 | 16.7 | 1.7 | 11 | 7 | 0 | 0 |
| Claude Haiku 4.5 | 49.7 | 17.3 | 7.6 | 57 | 3 | 1 | 0 |
| Qwen3.6-Plus | 46.4 | 17.9 | 0.0 | 61 | 6 | 2 | 0 |
Table 5: Diagnostic decomposition of failed episodes (full 897-set, centralized, image observation). Prog. (fail) = mean task progress on failed episodes (%); Steps (succ.) = mean executed steps on successful episodes; Inv. act. = invalid-action ratio (%); Dep/Occ/Aff/Ill = share of failed episodes exhibiting a dependency violation, occupancy conflict, affordance failure, or illegal-skill use (%), pooled over D1–D4 and weighted by each dimension’s failure count.
| Model | Time (s) | Steps | Planner calls |
|---|---|---|---|
| Proprietary (API) | |||
| GPT-5.6-sol | 123.8 | 18.4 | 19.2 |
| GPT-5.4-mini | 120.0 | 31.1 | 31.7 |
| Claude Opus 4.8 | 296.9 | 21.2 | 22.0 |
| Claude Haiku 4.5 | 215.2 | 26.0 | 26.7 |
| Qwen3.6-Plus | 106.7 | 26.1 | 26.9 |
Table 6: Per-episode compute cost (full 897-set, centralized, image observation). Time = mean wall-clock time per episode (s); Steps = mean executed action steps; Planner calls = mean planner (LLM) invocations per episode. Time reflects our evaluation hardware and API latency, not FLOPs, and is not directly comparable across proprietary and self-hosted open-weight models.
Figure 11: Successful planning for D1 task allocation. In Store Groceries , agents divide the independent grocery-storage and light-switch subtasks, avoid duplicate work, and jointly satisfy all goal predicates.
Figure 12: Successful planning for D2 sequential ordering. In Stock Cabinet , the team preserves the open–load–close dependency and issues the terminal close action only after all required placements.
Figure 13: Successful planning for D3 mutual exclusion. In Prepare Food , agents schedule access to the single shared Knife and complete their assigned slicing goals without resource collisions.
Figure 14: Successful planning for D4 handoff coordination. In Cross-Zone Handoff , producer and consumer agents relay objects through an intermediate transfer point before completing final placement.