Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams

Sahan Paliskara; Nattaput Namchittai; Andrew Lampinen

Figure 10: Scenario-by-scenario global outcomes for all formations across models for the API key environment. Episode counts: Appendix E . Whiskers are 95% confidence intervals. Solo is omitted. Crosses mark the three Opus 5 scenarios excluded due to API errors (stakes 130, cls 197, and duo 409). The scenario codes act as unique identifiers for the scenarios. We divide the groupings of scenarios by number of users using dashed lines. Across models, the coordinator clearly outperforms the teams in 8- and 16-person scenarios, and peer-to-peer teams struggle in many 4-person scenarios as well.

Figure 11: Share of peer-to-peer team members that ever act in the API key environment (at least one logged action attempt), averaged over episodes, by model and team size (4, 8, or 16 agents, under the bars). Main-text Figure 3 plots these values against the teams’ scores. Three Opus 5 scenarios are excluded due to API errors; episode counts: Appendix E .

Figure 12: API key mitigations. We group scenarios by their team sizes. Each bar represents the mean share of the optimal global outcome pooled across scenarios with 95% confidence intervals being the whiskers. The dashed line marks the coordinator. Below the dotted rule, mitigations are hatched: filled bars represent lead-only mitigations where only the lead can change jobs, while outlined bars are advisory mitigations in which any agent can mutate jobs. Full stack refers to using all four mitigations together. Episodes in which an agent’s model call failed four times in a row with a provider error (mostly a rate limit or overload) are excluded, in every arm and the control alike, which removes three Opus 5 control scenarios; episode counts: Appendix E .

Figure 13: We apply mediation analysis to the team sizes for the peer-to-peer teams in the scenarios in the API key environment. We consider the deltas between the 4- to 8-user teams (solid) and 4- to 16-user teams (hollow). The bars represent the coefficients of a least-squares fit of episode score on 8-user and 16-user indicators over every peer-to-peer episode, with no other predictor (none) or adding participation, start time, wasted share, or all three. Participation is the share of agents that ever act, start time the minute of the first action, and wasted share the share of tokens spent on jobs that never finished. Whiskers are 95% intervals from resampling the model’s scenarios (25; 22 for Opus 5) 2,000 times. Three Opus 5 scenarios are excluded due to API errors; episode counts: Appendix E .

Figure 14: Merge queue results of the two Claude models on three scenarios: oversub (one CI runner) and vol p1 and vol p3 (two runners, two profiles of CI times). Bars show the value merged before the freeze as a percentage of the best feasible plan, with 95% bootstrap intervals over episodes. The last two bars add a mitigation to the peer-to-peer team. Team + charter gives every agent a written charter, with no lead: treat the four PRs as one release, agree on a merge order early, order the PRs by value per minute of CI time, and hand the runner on when a PR cannot be ready in time. Team + lead + cut line adds a team lead and tells the lead and every agent the release manager’s priority order of the PRs; the lead’s first post drops the last PR in that order from the release. The prompts are given below. Episodes with infrastructure errors are excluded; episode counts: Appendix E .

Figure 15: Merge queue mitigation results. Rows of panels are models. Columns are the three scenarios, coded as in Figure 14 (volunteer profile 2 is not reported: no profile 2 episode reached the best feasible plan). A bar is welfare: the value of the pull requests merged before the freeze as a percentage of the scenario’s best feasible plan. Whiskers are 95% percentile bootstrap intervals over episodes, the number to the right of each bar is its value, and the dashed line marks the coordinator. Rows above the dotted rule are the four formations; the hatched rows r1 to r9 below it are the arms defined above; in r3 to r7 the plus signs name the clauses added to r2, not a cumulative order. Episodes in which a provider error ended an agent’s turn are excluded; episode counts: Appendix E . Only on oversub, and only for Claude Opus 5, does any arm beat the peer-to-peer team with a paired 95% interval above zero (r4 and r9; r8’s lower bound is near zero).

Model	Arm	oversub	vol p1	vol p3
Opus 5	Coordinator	79	98	100
	Peer-to-peer team	65	89	91
	Agents told the order	97	99	91
	+ stand-down (explicit prioritization)	83	80	79
	Lead and agents told the order	45	84	100
	+ cut line (team lead + cut line)	95	100	95

Appendix

Table 1: Merge queue mitigations: release value as a percentage of the best feasible plan. The coordinator, team, and crew charter rows are those of Figure 14 . Episodes in which a provider error ended one of the agents’ turns despite the harness’s retries are excluded; episode counts: Appendix E .

Model	Intervention	Scenario	Pairs	Difference	95% interval
Opus 5	Charter, no lead	oversub	15	+2.7	[-6.8, +12.0]
		vol p1	20	-11.4	[-18.5, -4.0]
		vol p3	19	-13.6	[-19.7, -7.9]
	Lead + full stack	oversub	14	-2.3	[-12.5, +8.0]
		vol p1	18	-10.6	[-21.6, -1.5]
		vol p3	20	-13.9	[-20.1, -8.3]

Appendix

Table 2: Merge queue: intervention minus coordinator on each scenario, in percentage points of the best feasible release value. The interventions are r9 (charter, no lead) and r8 (lead + full stack) of Figure 15 . Each difference pairs the same episode indices. Intervals are 95% percentile intervals. Episodes in which a provider error ended an agent’s turn are excluded (Appendix E ).

Figure 16: Clinic success rates per scenario per model per condition (assembled vs scattered). Success is measured as the joint success of both booking urgent patients by the deadline and booking the displaced patient. As part of development, each scenario was screened such that a Claude Sonnet 5 coordinator could get a joint success rate of 80% to ensure the benchmark was saturated by a single agent, and we could isolate measuring coordination. Each bar represents the share of the max joint success rate pooled over its episodes (20; 5 for the silent team). The solid bar represents the assembled version of the scenario while the dotted bar represents the scattered variant. The episode counts can be found in Appendix E .

		Missed	Booked on a	Wrong specialty,	Given an	Left with
Model	Facts	of 4,180	later day	same day	alternative	nothing
Opus 5	assembled	996	75.1	0.7	23.2	1.0
	scattered	1,034	57.2	0.8	34.6	7.4
Sonnet 5	assembled	545	10.3	20.4	53.0	16.3
	scattered	2,297	13.8	5.2	78.5	2.4

Appendix

Table 3: Exact values of Figure 7 (right panels), for the peer-to-peer team. Top: Describes what happened to urgent patients who missed their deadline, by model and condition. Bottom: Describes how many urgent patients of the scattered scenarios had all of the information that was relevant to them collected before either the last action that touched them or the end of the episode. We look both at all of them and the subset of them that failed. Episode counts: Appendix E .

Figure 17: Clinic mitigation stacks pooled on all 24 scenarios, Claude Opus 5 (left) and Claude Sonnet 5 (right). Bars are joint success: the share of urgent patients booked by the deadline with every named patient displaced for the booking also getting rebooked. Each bar averages the 12 scenarios of one information condition (left group: assembled facts; right group, dotted: scattered facts). The hatched bars right of each dotted rule are the stacks s1 to s3. The coordinator and team bars are the coordinator and the peer-to-peer team (the control) on episodes 1–10 of the main experiment, rescored with the same rule, so they can differ slightly from Figure 7 . Whiskers are 95% percentile intervals from 2,000 draws resampling episodes within each scenario. Episode counts: Appendix E .

Figure 18: Clinic instruction stacks on every scenario, Claude Opus 5 (left) and Claude Sonnet 5 (right); top: the 12 scenarios with assembled facts; bottom: the 12 with scattered facts. Each row is one scenario, labeled by the first drug of its family. Markers are joint success over the scenario’s episodes, as in Figure 17 ; whiskers are 95% percentile intervals from 2,000 draws resampling those episodes.

Figure 19: Personal assistant, request fulfillment per scenario, one row per model. A bar is the share of a scenario’s usable episodes whose committed order or booking honors the ask; an episode that commits nothing counts as unfulfilled (episode counts: Appendix E ). Whiskers are 95% Wilson intervals; a silent-team bar of height zero shows only its whisker.

Figure 20: Personal assistant mitigation arms. Left: Claude Opus 5; right: Claude Sonnet 5. Row labels abbreviate the arms listed above; the arms are hatched and sit below each group’s dotted rule, and teal marks the team-lead arm (m12). Each bar is a flip rate on failed peer-to-peer team episodes: the share that keep the user’s ask when re-run under the arm, not a fresh-sample success rate. The top group’s runs re-ran the whole set of 25 scenarios; the bottom group’s arms re-ran the episodes of a subset of 12 scenarios (11 for Sonnet 5), to save time. Each group has its own control (the same episodes re-run unchanged; dashed line): 45.5% and 41.9% for Opus 5, 30.3% and 26.1% for Sonnet 5. The top bar of each group is the coordinator’s rate of keeping the ask on the same scenario instances in the original run (over its admissible episodes, as below), the level the mitigations aim to reach. An asterisk marks a Holm-adjusted

p<0.01

in the paired McNemar test against the group’s control, adjusted over the ten arms of each model. Whiskers are 95% Wilson intervals over each bar’s admissible episodes (re-runs that committed and passed the scenario’s validity canaries). Re-runs in which a provider error ended an agent’s turn are excluded (episode counts: Appendix E ).

Figure 21: Ask in context and the guard’s effect in the personal assistant. To test the mitigations we took a sample of peer-to-peer team episodes that failed in the original run and re-ran each one twice from the same starting point: once unchanged (the control) and once with the mitigation (the arm). The top panels show the share of counted re-runs whose committing agent had the ask in context at the commit (the asker’s own agent committed, or the ask arrived by DM or by a message-history pull before the commit call), with 95% Wilson whiskers: the top four rows on the 21 scenarios with a DM log on both re-runs, the bottom eight on a subset of 12 of those (11 for Sonnet 5), each group with its own control (dashed line); the arms are hatched and sit below each group’s dotted rule, and teal marks the team-lead arm (m12). The bottom panels compare each arm with its control on the same episodes, using only episodes whose control re-run and arm re-run both count. The solid bar (“minus control”) is the arm’s gain in keeping the ask, in percentage points; the outlined bar (“adjusted for ask”) is the gain that remains once we account for whether the committing agent had the ask in context. Whiskers are 95% bootstrap intervals from 4,000 resamples of episodes within each scenario. The indicator counts only the DM route, so m11 and m13, which show a person’s note another way, can keep the ask without raising it, and it is measured after the arm acts, so the adjustment describes an association, not an identified causal path. Adjusting across episodes instead of within them leaves more (

+10

and

+14

points for Opus 5,

+17

and

+23

for Sonnet 5, m10 then m19). Re-runs in which a provider error ended an agent’s turn are excluded (episode counts: Appendix E ).

Model	Episodes	Any override	12+ edits	Max	Committed	Ask in context	Any fabrication
Opus 5	2,418	1,753 (72.5%)	115	55	2,106	1,369 (65.0%)	1,390 (57.6%)
Sonnet 5	2,374	1,757 (74.0%)	282	519	2,266	1,068 (47.1%)	1,266 (53.3%)
GPT-5.6-sol	1,220	787 (64.5%)	73	53	1,044	701 (67.1%)	—
GPT-5.6-terra	1,236	950 (76.9%)	97	58	1,206	633 (52.5%)	—

Appendix

Table 4: Personal assistant behavioral counts over all 25 scenarios, peer-to-peer teams only. Some episodes, as described in Appendix E , are excluded due to provider failures. An override is an executed pre-commit edit that puts the requesting user’s order or booking into a worse state that the scenario’s own contract scores as violating; 12+ edits counts episodes with at least twelve overrides, and Max is the most in one episode. Committed episodes have a checkout or booking; ask in context is the share of them in which the ask was in the committing agent’s context at the commit. A commit by the asker’s own agent counts as in context; a message sent before a commit may not yet be in the committing agent’s context. Fabrications are unsupported claims about a person made by an agent. These are verified using an Opus 5 classifier. Specifically, the model identifies the claims, a check then locates each claim in the transcript (to ensure it is not a hallucination), and a separate pass validates the located claims. Fabrication shares are over the episodes the judge labeled (Opus 5: 2,415 of 2,418); the judge was run on the Claude models only (—).

Scenario	Opus 5	Sonnet 5
anniversary pour	10.2	8.8
birthday thali floor	8.6	7.5
boarding call congee	6.9	10.7
fifteen line paste	6.8	2.1
sheet never had her	6.3	28.9
graduation banquet swap	4.8	36.6

Appendix

Table 5: Mean number of override edits across episodes for the 25 scenarios in the personal assistant environment.

Figure 22: Personal assistant with AGENTS.md in context as a mitigation for the teams: the peer-to-peer and silent teams in the run of record (hollow) and rerun with AGENTS.md in context (filled), beside the coordinator of the run of record (hollow squares). A value is the share of episodes whose committed order or booking honors the ask; an episode that commits nothing counts as unfulfilled, and usable episodes exclude canary exclusions and recorded provider failures. Left: pooled rates over all 25 and over the 9 safety scenarios; whiskers are 95% Wilson intervals. Right: each team cell (scenario, model, and team) rerun against its record cell; points above the dashed identity line rose. Every team row is higher with AGENTS.md in context, but the peer-to-peer team stays below the coordinator for both models on both scenario sets. Episode counts: Appendix E .

	Opus 5		Sonnet 5
	record	AGENTS.md	record	AGENTS.md
Episodes analyzed (count)	2,418	2,413	2,374	2,463
Episodes that checked out (count)	2,106	2,328	2,266	2,361
Episodes with an override (% of episodes)	1,753 (72.5%)	1,771 (73.4%)	1,757 (74.0%)	1,428 (58.0%)
ask restored by checkout (% of override episodes)	787 (44.9%)	871 (49.2%)	639 (36.4%)	819 (57.4%)
Episodes with 12+ override edits (count)	115	80	282	96

Appendix

Table 6: Personal assistant peer-to-peer teams on all 25 scenarios with an AGENTS.md that provides house rules in the context of the agents. We record how many episodes have specific behaviors. Specifically, we look at whether a cart committed at the end of an episode, whether any override happened, whether there were 12+ edits in the episode, how many edits of a user’s cart happened in the episode with the max number of these, whether the ask was in context the last time they were touched, how many episodes had empty rows, and whether a peer edited another’s. Rules 1 and 2 are the first two house rules in the food-order box on page 22 . Episodes with a recorded provider failure are excluded (episode counts: Appendix E ).

Environment	Model	Scenario(s)	Formation	Episodes per cell	Thinking
API key	All five models	all 25: 10 at 4 users, 8 at 8, 7 at 16	solo, coordinator, silent team, peer-to-peer team	20	Claude: not requested GPT: medium Qwen: minimal
	Qwen3.8-max	duo 409 (16 users)	peer-to-peer team	18	minimal
	Qwen3.8-max	pyr 154 (8 users)	peer-to-peer team	19	minimal
	Claude Opus 5	all 25, pooled per team size (4 / 8 / 16 users)	silent team	199 / 148 / 137 (1 / 12 / 3 excluded for provider errors)	not requested
	Claude Opus 5	all 25, pooled per team size (4 / 8 / 16 users)	peer-to-peer team	166 / 126 / 107 (34 / 34 / 33 excluded for provider errors, including every episode of stakes 130 at 4 users, cls 197 at 8 users and duo 409 at 16 users)	not requested
	Claude Sonnet 5	all 25, pooled per team size (4 / 8 / 16 users)	solo	200 / 159 / 138 (0 / 1 / 2 excluded for provider errors)	not requested

Appendix

Table 7: Formation experiments: the italic first row of each environment is its default, and the rows beneath list only departures from it. Episodes per cell are the episodes of one model, scenario, and formation used after the paper’s validity rules. An API key episode is excluded for provider errors when one of its agents’ model calls failed four or more times in a row with a provider error. The personal assistant’s provider failures are episodes that did not commit and whose agent session ended in a dead turn or an unrecovered run of overload errors.

Environment	Model	Scenario(s)	Arm	Episodes per cell	Thinking
API key	Claude Opus 5, Claude Sonnet 5	all 25	10 team-lead arms, lead-only and advisory: lead alone, + value, + defer, + release, full stack; control (peer-to-peer team) and coordinator reused from Table 7	20	not requested
	Claude Opus 5	duo 387 (16 users)	advisory: + release	18 (incomplete at the freeze)	not requested
	Claude Opus 5	skl 350 (16 users)	advisory: lead alone	18 (incomplete at the freeze)	not requested
	Claude Opus 5	skl 414 (16 users)	advisory: + defer	18 (incomplete at the freeze)	not requested
	Claude Opus 5	skl 463 (16 users)	advisory: + defer	19 (incomplete at the freeze)	not requested
	Claude Sonnet 5	stakes 480 (16 users)	advisory: + value	18 (incomplete at the freeze)	not requested

Appendix

Table 8: Mitigations, laid out as Table 7 : the italic first row of each environment is the default and the rows beneath list only departures. Episodes per cell are those used in the paper for one model, scenario, and arm.

	API key	Clinic	Merge queue	Personal assistant
Runs in	Inspect, with pi’s system prompt and tools; the harness answers tool calls, and Claude Opus 5 answers those it cannot and unscripted messages to users	Inspect, with a clinic console prompt and only the clinic tools; the harness answers every call and plays the scripted callers	One pi 0.77.0 process per agent, with real Git clones and tests and a gh tool; Claude Sonnet 5 writes a user’s reply when the user is addressed	One pi 0.77.0 process per agent, with only the environment’s tools
Scheduling	One agent acts at a time; its turn lasts until it replies in text or sends a message. The harness then wakes an agent with an unread message or a newly due user message, otherwise the next agent in a fixed order	As in API key, but standing by also ends a turn, and the desk whose caller has waited longest comes before the fixed order	Rounds: each agent takes one turn per round, one after another, in an order reshuffled every round; a turn lasts until the agent stops	All agents run at the same time; a turn starts with the opening message or a DM and lasts until the agent stops
Clock	One simulated clock for all agents, from the start of the episode to the deadline at its end. Turns, messages and reads take no time; a job-changing call adds 1 minute and labctl_wait the minutes asked for (at most 60). When no agent has just changed a job and no message is waiting, the clock jumps to the next scheduled event (at most 90 minutes)	One simulated clock for all desks, from the start to the end of the episode. Only standing by moves it, to the next scheduled event or by 20 minutes, whichever is sooner; an idle desk that the harness wakes stands by automatically	Simulated minutes, frozen during a round; after each round the clock and the CI runs advance 5 minutes	Real time; the order or booking window closes 21 minutes after the opening messages
Stopping	At the end of the episode, once every agent has answered its users’ end-of-episode question; an agent still holding an undelivered message or result gets up to two more turns	At the end of the episode. The coordinator gets up to two more turns as in API key; to account for the extra communication in a team, the teams’ wrap-up continues until every call is delivered, waking any desk that has not yet taken a turn	At the freeze (minute 105 in oversub, 100 in vol p1 and vol p3); each agent then gets one final turn to answer its users	At the first commit (each agent then gets one final turn), when no agent is working and no DM is waiting, or when the window closes (turns under way finish, and a commit they make counts)
Limits	Per episode: 800, 1,000 or 1,400 scheduler steps at 4, 8 or 16 users (never reached) and a wall-clock limit	Per episode: a scheduler-step cap (300 for the coordinator, higher for the teams) and a wall-clock limit	One turn per agent per round; a turn is cut off after 30 minutes of real time	A turn is cut off after 30 minutes of real time
Messages	SendMessage ends the sender’s turn; the message opens the recipient’s next turn	As in API key; notes and bookings are visible to all desks as soon as they are made	PR and issue comments, readable with gh at any time and shown in a feed that opens each agent’s next turn	A DM opens the recipient’s next turn once its current turn ends; dm_history shows the thread

Appendix

Table 9: How each environment runs its agents. A scheduler step wakes one agent once. In the API key and clinic environments, an episode cut off by a step or wall-clock limit is not scored. Agents have no token budget, and in the API key and clinic environments no turn limit of their own.

API key
		Original	Rank	Random values
	Scenarios	values	values	in the same order
		coord./team/tie	coord./team/tie	median [range]
Claude Opus 5	22	22 / 0 / 0	22 / 0 / 0	22 [22–22]
Claude Sonnet 5	25	20 / 5 / 0	20 / 5 / 0	20 [19–22]
GPT-5.6-sol	25	20 / 5 / 0	20 / 5 / 0	20 [19–21]

Appendix

Table 10: In the API key and merge queue environments, an episode’s score is the value of the jobs or PRs it completes, priced with values we assigned; the exact numbers are somewhat arbitrary, but their priority order, which is what the agents act on, is not. We therefore rescore the same episodes with other values in the same order: each job’s or PR’s priority rank (1 for the lowest priority), and 1,000 random value sets per scenario. Each triple counts the scenarios in which the coordinator scores above / below / equal to the peer-to-peer team; the last column is the median [2.5–97.5% range] of the first count over the random value sets. Rank values change the winner in one scenario (GPT-5.6-terra, API key), and the random ranges stay within 2 of the original counts. Three Opus 5 scenarios are excluded due to API errors.

Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams

Abstract

Figures & tables

Appendix

Explore similar work

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents