TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information

Chanwoo Park; Jaywoong Jeong; Zachary Wojtowicz; Yubin Kim; Taekyun Lee; Kaiqing Zhang; Asuman Ozdaglar

cs.AIOct 27, 2026

TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information

Authors: Chanwoo Park, Jaywoong Jeong, Zachary Wojtowicz, Yubin Kim, Taekyun Lee, Kaiqing Zhang, Asuman Ozdaglar

Organizations: MIT · KAIST · UT Austin · University of Maryland

Abstract

Many real-world decisions require teams to integrate information distributed across participants. Yet AI agents are still primarily trained and benchmarked in isolation. We introduce TeamGym, an environment for evaluating and training language model-based agents on team decision-making problems constructed from real-world legal, finance, medical, and software engineering corpora. TeamGym introduces a method to identify a minimal set of facts, each element of which is necessary and all of which are collectively sufficient for a correct final decision, which we call pivotal information. This enables TeamGym to both control the distribution of pivotal information and track how well agents identify and integrate pivotal information when making team decisions. Examining 8 language models on 12 source corpora, we find that current AI agents often fail to effectively share information: Distributing pivotal information across participants lowers success rate by 22.9 percentage points on average relative to giving the same information to a single model (a communication gap). We also document that participants do not effectively retrieve decision-relevant information on their own: A team's success rate is lowered by a further 19.9 points when participants must identify and extract pivotal information from source material rather than receiving it directly (a retrieval gap). To help remedy these deficits, we use TeamGym to train policies through reinforcement learning on team decision loss. Across two policy-training experiments, we find that training improves final-decision quality and induces emergent coordination in participation, speaker selection, and communication, even though the reward evaluates only the final decision.

Figures & tables

Domain	Source corpora	Decisions
Legal	CUAD, MAUD, eCFR Casebook, MSLR, Multi-LexSum, SARA	Contract profiles, merger reviews, compliance and enforcement, legal and procedural conclusions, and tax calculations
Finance	FinDVer, FinLongDocQA	SEC-filing findings, calculations, comparisons, and classifications
Medical	MLCR, EvidenceBench	Evidence-grounded medical reviews and biomedical answers
Software engineering	SWE Atlas–Codebase QnA, DeepSWE	Codebase QA and multi-turn, long-horizon repository-level code repair with functionally evaluated patches

Table 1: Source corpora and decisions. The problem instances constructed from the source corpora use different source materials and decision objectives across four domains.

Fig. 2 : Communication structures (

\mathcal{C}

). The three structures on the left have no coordinator and use a separate final decision maker. In the two structures on the right, a coordinator controls interaction and makes the final decision. The rule below each panel specifies who may send messages in each communication round.

Fig. 3 : PIU construction and allocation. Candidate evidence units are grounded in the source material, checked for joint sufficiency and leave-one-out necessity, and retained as an inclusion-minimal pivotal-information set. The resulting PIUs can then be allocated across participants while preserving the decision task and reference decision.

Fig. 4 : Five PIU-distribution conditions. Each condition distributes the

\mathcal{I}^{\min}_{q}

across the initial local and common observations without changing

q

d^{\mathrm{ref}}

. Colored boxes denote distinct PIUs, and

\varnothing

denotes a participant assigned no PIU. The common-observation panel marks information initially visible to the full team. Not every problem instance admits every condition.

Fig. 5 : Three evaluation conditions and aggregate performance gaps. Single-Agent All-PIU provides all pivotal information to one model; Multi-Agent PIU-Only distributes those PIUs across participants; and Multi-Agent Full-Context embeds the same PIUs in participants’ source materials. The communication gap compares the first two conditions, and the retrieval gap compares the two multi-agent conditions.

Legal			Finance, medical, and software engineering
Source corpus	Constructed problem instances	Evaluated problem instances	Domain	Source corpus	Constructed problem instances	Evaluated problem instances
CUAD	538	30	Finance	FinDVer	389	30
MAUD	557	30	Finance	FinLongDocQA	249	30
eCFR Casebook	18	18	Medical	MLCR	29	29
MSLR	11	11	Medical	EvidenceBench	45	45
Multi-LexSum	29	29	Software engineering	SWE Atlas–Codebase QnA	32	32

Table 2: Problem-instance counts across four domains and twelve source corpora.

Fig. 6 : Cross-domain episode success rates. Each domain panel compares the three evaluation conditions for each model; the model-average row weights models equally. The decrease from Single-Agent All-PIU to Multi-Agent PIU-Only defines the communication gap, and the further decrease to Multi-Agent Full-Context defines the retrieval gap. Error bars indicate standard errors. Labels abbreviate Claude Sonnet 4.5 and Haiku 4.5 as Sonnet and Haiku; Gemma labels refer to Gemma 3. Appendix E reports source-corpus results, evaluation coverage, and failure handling.

Measurement	Base policies	Two-stage training	Joint two-policy training
Every participant queried	80.6	66.7	100.0
Mean words per query	14.0	48.0	12.5
Question copied in first query	0.0	88.6	0.0
Later queries reusing preceding-reply terms	3.2	42.9	13.5

Table 6: Coordinator-query behavior. CUAD-derived validation (

n=36

); percentages except mean words per query. Behavior rates use eligible queries; reuse of preceding-reply terms is measured lexically.

Table 10

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

	LLM-judged score	Final-decision completion	Same final- decision policy	PIUs appearing in the communication transcript	Exactly repeated public messages
FinDVer validation	21.85 $\rightarrow$ 41.08	51.3 $\rightarrow$ 84.6	34.10 $\rightarrow$ 46.72	38.0 $\rightarrow$ 53.9	18.9 $\rightarrow$ 50.9
FinLongDocQA test	33.50 $\rightarrow$ 41.13	65.0 $\rightarrow$ 92.5	49.60 $\rightarrow$ 44.82	63.1 $\rightarrow$ 63.1	23.1 $\rightarrow$ 48.6

Appendix

Table 10: Communication-trajectory analysis. Arrows compare the base and trained policies. All entries are percentages. “Same final-decision policy” is the LLM-judged score obtained by applying one common GPT-5.5 final-decision policy to every communication transcript.

Source corpus	Underlying source materials	Reference decision	Problem family
CUAD	Expert-annotated commercial contracts	Constructed alternative	Contract-profile decision
MAUD	Expert-curated merger-agreement provisions	Constructed alternative	Merger-review decision
eCFR Casebook	Regulations and controlled transaction facts	Written conclusion	Compliance analysis
MSLR	Anonymized enforcement case files	Final sanction record	Enforcement conclusion
Multi-LexSum	Documents from a litigation collection	Case-grounded decision	Procedural analysis
SARA	Tax facts and supplied statutory provisions	Numeric result	Tax calculation

Appendix

Table 11: Legal problem families and source corpora.

Fig. 8 : Problem-instance and PIU-distribution statistics in the legal evaluation. In the PIU-count and PIU-length panels, each point is one problem instance. In the participant-count and source-material-length panels, each point is one PIU distribution. Boxes show the median and interquartile range; token-length panels use logarithmic axes.

Source corpus	Problem instances	PIU distributions	Participants/ distribution	PIUs	Source-material tok./ distribution	Source-material tok./ participant	PIU tok./ unit
CUAD	30	42	3.0	2.0	15,644.9	5,215.0	116.4
MAUD	30	53	3.5	3.4	70,859.8	20,759.0	174.8
eCFR Casebook	18	61	6.7	6.3	68,383.3	10,270.3	96.3
MSLR	11	20	6.3	6.0	35,948.0	5,688.9	141.7
Multi-LexSum	29	113	10.4	10.4	15,621.4	1,535.4	69.2
SARA	45	225	8.9	8.7	97,936.4	10,969.9	64.3

Appendix

Table 12: Mean problem-instance and PIU-distribution statistics for the legal evaluation. Problem-instance measures use all 163 problem instances. PIU-distribution measures use 514 distributions. For measures that depend on PIU distribution, we first average within problem instance and then across problem instances. Source-material lengths reflect the material provided after applying the initial-local-observation token budget of 40,000 tokens.

CUAD label family	Information in annotated spans	Decision objective
Source Code Escrow	deposit, release trigger, post-release use right, update duty, cost	source-code continuity protection strength
ROFR/ROFO/ROFN	covered opportunity, notice, exercise period, matching or negotiation right, expiration	transaction restriction strength
Insurance	coverage type, coverage limit, certificate, cancellation notice, additional insured, remedy	insurance burden or protection level
Audit Rights	recordkeeping, records review, facility or system inspection, notice, frequency, cost	audit burden level
License Grant	grant, field of use, territory, duration, sublicense, use restriction	license sufficiency for intended use
Minimum Commitment	amount, period, purchase or revenue metric, shortfall remedy, exceptions	commercial commitment materiality

Appendix

Table 13: CUAD label families used in problem-instance construction. The decision objective specifies a legal profile supported by multiple annotated spans rather than the original CUAD label.

CUAD label family	Example participant descriptions
Source Code Escrow	deposit reviewer; release-condition reviewer; use-right reviewer; update/cost reviewer
ROFR/ROFO/ROFN	covered-opportunity reviewer; notice reviewer; exercise-period reviewer; post-expiration reviewer
Insurance	coverage-type reviewer; coverage-limit reviewer; certificate/notice reviewer; special-coverage reviewer
Audit Rights	records reviewer; access reviewer; operations-inspection reviewer; process/cost reviewer

Appendix

Table 14: Example participant descriptions for CUAD problem instances.

MAUD category	Information in provision excerpts	Decision objective
Deal protection and related provisions	no-shop limits, fiduciary-out conditions, matching periods, superior-proposal conditions, and termination-fee provisions	board-process or termination-trigger profile
Conditions to closing	representation bring-down timing and standard, materiality treatment, and fundamental-representation treatment	closing-condition risk profile
Material adverse effect	base definition, carveouts, disproportionate-impact treatment, and pandemic or market exceptions	material-adverse-effect risk-allocation profile
Operating and efforts covenant	ordinary-course covenant, negative covenants, buyer consent rights, regulatory-efforts language, and emergency exceptions	pre-closing control profile

Appendix

Table 15: MAUD categories used in problem-instance construction. Each problem instance uses provision excerpts from one agreement and one category.

Ordering	Mean adjacent cosine similarity	Episode success (%)	LLM-judged score (%)	Answer-component coverage (%)
Seeded random	.449	13.8 $\pm$ 6.4	65.2 $\pm$ 4.3	80.7 $\pm$ 3.4
Maximum similarity	.598	27.6 $\pm$ 8.3	71.9 $\pm$ 3.5	87.4 $\pm$ 2.8

Appendix

Table 16: Sensitivity to the ordering used to construct Pairwise PIU overlap distributions for Multi-LexSum. Values average over the same 29 problem instances. Smaller

\pm

terms are standard errors across problem instances.

Participant descriptions	Episode success rate (%)	LLM-judged score (%)	Answer-component coverage (%)
Constructed from participants’ assigned PIUs	24.1 $\pm$ 8.1	77.6 $\pm$ 1.9	84.8 $\pm$ 2.4
Same descriptions permuted across participants	0.0 $\pm$ 0.0	46.9 $\pm$ 4.8	67.2 $\pm$ 3.6

Appendix

Table 17: Sensitivity to the assignment of participant descriptions for Multi-LexSum. Values average over the same 29 problem instances. Smaller

\pm

terms are standard errors across problem instances.

Source corpus	Underlying source materials	Reference decision	Problem family
FinDVer	SEC filings with source questions and annotated contexts	Written filing-review answer	Filing-review reasoning
FinLongDocQA	Company-year annual reports with source questions and answers	Written annual-report answer	Annual-report reasoning

Appendix

Table 18: Finance problem families and source corpora.

PIU-distribution condition	FinDVer	FinLongDocQA
One PIU per participant	30	30
Pairwise PIU overlap	26	23
Extra participant without a PIU	30	28
One common PIU	28	23
All-but-one PIU concentration	30	29
PIU distributions, total	144	133

Appendix

Table 19: Finance problem-instance counts by PIU-distribution condition.

Statistic	FinDVer	FinLongDocQA
Source questions in the complete release	2,400	7,527
Source questions considered during construction	700	7,527
Available underlying source materials	255 filings	1,329 company-years
Constructed problem instances	389	249

Appendix

Table 20: Finance source-corpus and problem-instance counts.

Fig. 9 : Problem-instance and PIU-distribution statistics in the finance evaluation. In the PIU-count and PIU-length panels, each point is one problem instance. In the participant-count and source-material-length panels, each point is one PIU distribution. Boxes show the median and interquartile range; token-length panels use logarithmic axes.

Source corpus	Problem instances	PIU distributions	Participants/ distribution	PIUs	Source-material tok./ distribution	Source-material tok./ participant	PIU tok./ unit
FinDVer	30	144	4.5	5.1	44,038.1	11,887.5	271.3
FinLongDocQA	30	133	3.1	3.3	91,711.8	31,168.6	546.0
All source corpora	60	277	3.8	4.2	67,874.9	21,528.1	408.6

Appendix

Table 21: Mean problem-instance and PIU-distribution statistics for the finance evaluation. Problem-instance measures use the 60 fixed evaluation problem instances. PIU-distribution measures use 277 distributions. For measures that depend on PIU distribution, we first average within problem instance and then across problem instances. Source-material lengths reflect the material provided after participant-level leakage removal and the 40,000-token initial-local-observation cap.

Source corpus	Underlying source materials	Reference decision	Problem family
MLCR	Case-level visit-summary documents with source questions and answers	Written medical-record review	Medical-record reasoning
EvidenceBench	Biomedical-paper sentence pools with hypotheses and evidence annotations	Written biomedical-evidence answer	Biomedical-evidence reasoning

Appendix

Table 22: Medical problem families and source corpora.

PIU-distribution condition	MLCR	EvidenceBench
One PIU per participant	29	45
Pairwise PIU overlap	19	36
Extra participant without a PIU	28	45
One common PIU	19	36
All-but-one PIU concentration	19	45
PIU distributions, total	114	207

Appendix

Table 23: Medical problem-instance counts by PIU-distribution condition.

Statistic	MLCR	EvidenceBench
Source questions or hypotheses in the construction pool	205 source questions	293 test-split hypotheses
Available underlying source materials	1,071 visit-summary documents	293 biomedical-paper sentence pools
Evaluated problem instances	29	45
Evaluated PIU distributions	114	207

Appendix

Table 24: Medical source-corpus and problem-instance counts.

Fig. 10 : Problem-instance and PIU-distribution statistics in the medical evaluation. In the PIU-count and PIU-length panels, each point is one problem instance. In the participant-count and source-material-length panels, each point is one PIU distribution. Boxes show the median and interquartile range; token-length panels use logarithmic axes.

Source corpus	Problem instances	PIU distributions	Participants/ distribution	PIUs	Source-material tok./ distribution	Source-material tok./ participant	PIU tok./ unit
MLCR	29	114	3.2	3.2	38,119.9	13,080.7	208.5
EvidenceBench	45	207	4.0	4.0	5,678.0	1,727.4	62.7
All source corpora	74	321	3.7	3.7	18,391.7	6,176.7	119.8

Appendix

Table 25: Mean problem-instance and PIU-distribution statistics for the medical evaluation. Problem-instance measures use the 74 fixed evaluation problem instances. PIU-distribution measures use 321 distributions. For measures that depend on PIU distribution, we first average within problem instance and then across problem instances. Source-material lengths reflect the material provided after participant-level leakage removal and the 40,000-token initial-local-observation cap.

Source corpus	Problem family	Evaluation criteria	Evaluated problem instances
SWE Atlas Codebase QnA	Multi-file and runtime code comprehension	Every answer component is complete and correct	32
DeepSWE	Long-horizon repository-level code repair	Official repository verifier passes	12

Appendix

Table 26: Software-engineering source corpus and evaluation status.

Fig. 11 : Software-engineering problem-instance and PIU-distribution statistics. In the PIU-count and PIU-length panels, each point is one problem instance. In the participant-count panel, each point is one PIU distribution; in the source-material-length panel, each point is one problem instance. SWE Atlas includes its five valid allocation modes; DeepSWE includes the 12 selected functional one-each allocations. Boxes show the median and interquartile range; token-length panels use logarithmic axes.

Source corpus	Problem instances	PIUs/ instance	Code/source units/instance	Runtime observations/instance	Full source-material tok./instance	PIU tok./ unit
SWE Atlas	32	7.3	236.5	0.0	261,951.8	130.2
DeepSWE	12	3.6	16.8	7.2	1,140,635.7	3570.6

Appendix

Table 27: Software-engineering structural statistics.

Source corpus	Leave-one-out pivotality	Participant-level source-material check
MAUD	64/67 (95.5%) [89.5, 100.0]	18/20 (90.0%) [69.9, 97.2]
FinLongDocQA	65/67 (97.0%) [92.3, 100.0]	20/20 (100.0%) [83.9, 100.0]

Appendix

Table 28: Independent validation of two representative PIU-construction regimes. Values are passing judgments or instances over the corresponding validation denominator, with percentages and 95% intervals in brackets. Instance- level intervals are Wilson intervals; leave-one-out intervals use an instance-cluster bootstrap with 10,000 resamples.

Domain	Source corpus	Problem instances	Models	Single-Agent All-PIU	Multi-Agent PIU-Only	Multi-Agent Full-Context
Legal	CUAD	30	8	90.4 $\pm$ 1.7	72.5 $\pm$ 2.7	62.9 $\pm$ 2.8
	MAUD	30	8	88.3 $\pm$ 1.8	67.9 $\pm$ 2.2	60.0 $\pm$ 2.7
	eCFR Casebook	18	8	88.9 $\pm$ 2.2	47.9 $\pm$ 3.5	18.1 $\pm$ 2.5
	MSLR	11	8	57.9 $\pm$ 4.0	45.5 $\pm$ 4.6	8.0 $\pm$ 2.6
	Multi-LexSum	29	8	62.5 $\pm$ 2.3	30.2 $\pm$ 2.3	20.2 $\pm$ 2.1
	SARA	45	8	63.6 $\pm$ 1.9	37.2 $\pm$ 2.0	8.1 $\pm$ 1.4

Appendix

Table 29: Evaluation results by source corpus. Values are episode success rates in percent. Models receive equal weight within each source corpus. The Models column gives scheduled coverage; unavailable estimates are omitted condition by condition. Smaller

\pm

terms are binomial standard errors propagated through each fixed-model average.

Domain	Source corpus	Model	Problem instances	Single-Agent All-PIU	Single-Agent Full-Context	Multi-Agent PIU-Only	Multi-Agent Full-Context
Legal	CUAD	GPT-5.5	30	96.7 $\pm$ 3.3	86.7 $\pm$ 6.2	93.3 $\pm$ 4.6	90.0 $\pm$ 5.5
		Claude Sonnet 4.5	30	100.0 $\pm$ 0.0	90.0 $\pm$ 5.5	53.3 $\pm$ 9.1	43.3 $\pm$ 9.0
		Claude Haiku 4.5	30	90.0 $\pm$ 5.5	93.3 $\pm$ 4.6	70.0 $\pm$ 8.4	60.0 $\pm$ 8.9
		Gemma 3 4B IT	30	60.0 $\pm$ 8.9	30.0 $\pm$ 8.4	50.0 $\pm$ 9.1	40.0 $\pm$ 8.9
		Gemma 3 12B IT	30	90.0 $\pm$ 5.5	53.3 $\pm$ 9.1	70.0 $\pm$ 8.4	36.7 $\pm$ 8.8
		Gemma 3 27B IT	30	93.3 $\pm$ 4.6	50.0 $\pm$ 9.1	80.0 $\pm$ 7.3	70.0 $\pm$ 8.4

Appendix

Table 30: Evaluation results by model and source corpus. Values are episode success rates in percent. Smaller

\pm

terms are binomial standard errors over each condition’s episode-success denominator. For the three primary conditions, confirmed context-window failures are counted as unsuccessful episodes, while provider-capacity and other infrastructure failures remain unavailable. Single-Agent Full-Context excludes context-window, infrastructure, and judge failures, while output failures are counted as unsuccessful episodes. For legal Single-Agent Full-Context, serving-context rejections and other evaluation errors are excluded from this denominator, while structured-interface failures at the final-decision stage are included as unsuccessful episodes. When fewer than the fixed inputs contribute to an estimate, its denominator is shown as

n/N

. An em dash denotes that no comparable estimate is available. Appendix E.3 reports the corresponding coverage and failure counts.

Evaluation condition	Strict success	Valid final decision, unsuccessful	Structured-interface failure	Success given a valid final decision
Single-Agent All-PIU	982/1,304 (75.3%)	318/1,304 (24.4%)	4/1,304 (0.3%)	75.5%
Multi-Agent PIU-Only	650/1,304 (49.8%)	531/1,304 (40.7%)	123/1,304 (9.4%)	55.0%
Multi-Agent Full-Context	404/1,304 (31.0%)	747/1,304 (57.3%)	153/1,304 (11.7%)	35.1%

Appendix

Table 31: Outcome decomposition for the primary legal evaluation. Structured-interface failures remain unsuccessful episodes in the reported episode success rate. The final column is a diagnostic conditional rate computed only over episodes that produce a valid final decision.

Domain	Model	Planned	Evaluated, $n$ (%)	Context unavailable, $n$ (%)	Output failures, $n$ (%)	Other failures, $n$ (%)
Legal	GPT-5.5	163	163 (100.0)	0 (0.0)	0 (0.0)	0 (0.0)
Legal	Claude Sonnet 4.5	163	156 (95.7)	0 (0.0)	7 (4.3)	0 (0.0)
Legal	Claude Haiku 4.5	163	161 (98.8)	0 (0.0)	2 (1.2)	0 (0.0)
Legal	Gemma 3 4B IT	163	147 (90.2)	16 (9.8)	0 (0.0)	0 (0.0)
Legal	Gemma 3 12B IT	163	147 (90.2)	16 (9.8)	0 (0.0)	0 (0.0)
Legal	Gemma 3 27B IT	163	147 (90.2)	16 (9.8)	0 (0.0)	0 (0.0)

Appendix

Table 32: Single-Agent Full-Context evaluation counts by domain and model. The table covers the 11 source corpora other than DeepSWE, for which this condition was not run. Percentages in parentheses use all fixed problem-instance–model pairs in each domain–model row. Context-window, infrastructure, and judge failures are excluded from the episode-success denominator; output failures are included as unsuccessful episodes.

Participant model	Final decision maker	$E(T)$	$E(T)+S(T)$	Effect	95% CI
Qwen3-32B	Qwen3-32B	54.4%	69.0%	$+14.6$	$[7.0,22.2]$
Qwen3-32B	GPT-5.5	69.6%	75.3%	$+5.7$	$[1.9,10.1]$
Qwen3-235B-A22B	Qwen3-235B-A22B	60.1%	77.8%	$+17.7$	$[10.1,25.3]$
Qwen3-235B-A22B	GPT-5.5	78.5%	86.7%	$+8.2$	$[3.2,13.3]$

Appendix

Table 33: Effect of correct synthesis on episode success. Effects and paired 95% bootstrap confidence intervals are in percentage points.

Participant model	Final decision maker	Effect	95% CI
Qwen3-32B	Qwen3-32B	$+2.5$	$[-1.3,6.3]$
Qwen3-32B	GPT-5.5	$+1.3$	$[-2.5,5.1]$
Qwen3-235B-A22B	Qwen3-235B-A22B	$+5.7$	$[1.9,10.1]$
Qwen3-235B-A22B	GPT-5.5	$+7.6$	$[3.2,12.7]$

Appendix

Table 34: Effect of removing error uptake. Effects and paired 95% bootstrap confidence intervals are in percentage points.

Evaluation condition	PIU-distribution condition	Broadcast query	Round robin	Current-speaker selection	Simultaneous responses
Multi-Agent PIU-Only $N=50,438$ joint $p<.001$	Pairwise PIU overlap	$4.5$ $\pm$ 1.4 *	$4.4$ $\pm$ 1.5 *	$4.5$ $\pm$ 1.9 *	$1.2$ $\pm$ 1.5
	Extra participant without a PIU	$0.6$ $\pm$ 1.5	$1.8$ $\pm$ 1.3	$-0.3$ $\pm$ 1.6	$-1.1$ $\pm$ 1.6
	One common PIU	$-0.6$ $\pm$ 1.5	$1.3$ $\pm$ 1.5	$2.7$ $\pm$ 1.8	$-3.3$ $\pm$ 1.5 *
	All-but-one PIU concentration	$-1.1$ $\pm$ 1.6	$4.3$ $\pm$ 1.5 *	$6.5$ $\pm$ 2.1 *	$-0.4$ $\pm$ 1.6
Multi-Agent Full-Context $N=48,499$ joint $p<.001$	Pairwise PIU overlap	$0.4$ $\pm$ 1.2	$2.3$ $\pm$ 1.2	$-1.5$ $\pm$ 1.5	$0.9$ $\pm$ 1.3
	Extra participant without a PIU	$-1.7$ $\pm$ 1.3	$-0.2$ $\pm$ 1.2	$-2.3$ $\pm$ 1.3	$-0.6$ $\pm$ 1.1

Appendix

Table 35: PIU-distribution–communication-structure interactions across the evaluated problem families. Entries are average interaction estimates in episode-success percentage points; smaller

\pm

terms are problem-instance-clustered standard errors. Rows and columns are relative to One PIU per participant and Coordinator-selected speaker, respectively. Yellow cells and asterisks mark 95% confidence intervals excluding zero.

Evaluation condition	PIU-distribution condition	Broadcast query	Round robin	Current-speaker selection	Simultaneous responses
Multi-Agent PIU-Only $N=50,438$ joint $p<.001$	Pairwise PIU overlap	$4.8$ $\pm$ 1.5 *	$4.7$ $\pm$ 1.5 *	$4.6$ $\pm$ 1.9 *	$1.6$ $\pm$ 1.5
	Extra participant without a PIU	$0.6$ $\pm$ 1.5	$1.9$ $\pm$ 1.3	$-0.3$ $\pm$ 1.6	$-1.0$ $\pm$ 1.6
	One common PIU	$-0.5$ $\pm$ 1.5	$1.5$ $\pm$ 1.5	$2.7$ $\pm$ 1.8	$-2.9$ $\pm$ 1.6
	All-but-one PIU concentration	$-1.1$ $\pm$ 1.6	$4.2$ $\pm$ 1.5 *	$6.3$ $\pm$ 2.1 *	$-0.3$ $\pm$ 1.6
Multi-Agent Full-Context $N=48,499$ joint $p<.001$	Pairwise PIU overlap	$0.4$ $\pm$ 1.2	$2.4$ $\pm$ 1.2 *	$-1.5$ $\pm$ 1.6	$1.0$ $\pm$ 1.3
	Extra participant without a PIU	$-1.8$ $\pm$ 1.3	$-0.2$ $\pm$ 1.2	$-2.3$ $\pm$ 1.2	$-0.7$ $\pm$ 1.1

Appendix

Table 36: Robustness analysis with problem-instance fixed effects. Entries are average interaction estimates in episode-success percentage points; smaller

\pm

terms are problem-instance-clustered standard errors. Columns replace Coordinator-selected speaker, and the four displayed PIU-distribution rows replace One PIU per participant. Light yellow cells and asterisks mark 95% confidence intervals that exclude zero.

PIU distribution	Coordinator-selected speaker	Round robin	Difference
One PIU per participant	62.7%	80.5%	$+17.8$
All-but-one PIU concentration	36.4%	67.8%	$+31.4$
Interaction (95% CI)			$+13.6$ $[0.8,26.3]$

Appendix

Table 37: Matched finance communication comparison. Episode success rates pool the two Qwen3 models across 59 problem instances. Effects are in percentage points.

Coordinator query condition	Interaction (95% CI)	Change (95% CI)
Original query without additions	$+15.5^{*}$ $[3.4,27.4]$	Reference
Original query with additions	$+5.9$ $[-5.1,16.4]$	$-9.6^{*}$ $[-16.9,-2.3]$
Query removed without additions	$-5.4$ $[-14.7,3.7]$	$-20.9^{*}$ $[-30.2,-11.3]$
Query removed with additions	$-9.3$ $[-20.1,0.8]$	$-24.9^{*}$ $[-34.7,-14.7]$

Appendix

Table 38: Finance communication-structure interactions with Round robin held fixed. The original Qwen3 Round robin allocation difference is

-12.7

points. Interactions subtract the coordinator allocation difference from this fixed reference. Changes compare each condition with the original query without additions. Entries are percentage points with 95% paired problem-instance bootstrap intervals; yellow and ∗ indicate statistical significance.

	Original coordinator query		Coordinator query removed
PIU distribution	Without additions	With additions	Without additions	With additions
One PIU per participant	65.5%	69.2%	71.5%	74.3%
All-but-one PIU concentration	37.3%	50.6%	64.1%	70.9%

Appendix

Table 39: Finance query interventions with fixed speaker selection. Episode success rates pool the two Qwen3 models, averaging three repetitions. “With additions” appends public PIU category labels corresponding to assigned PIUs and an instruction to report supporting evidence for each category. “Coordinator query removed” replaces the original task-specific question with “It is your turn to contribute,” while retaining speaker selection.

Comparison	Interaction	95% CI
Add labels and instruction to original coordinator query	$+9.6^{*}$	$[2.3,16.9]$
Remove coordinator query without additions	$+20.9^{*}$	$[11.6,30.2]$
Remove coordinator query with additions	$+15.3^{*}$	$[7.9,22.6]$

Appendix

Table 40: Finance intervention interactions across PIU distributions. Each interaction subtracts the intervention effect under One PIU per participant from that under All-but-one PIU concentration, pooling the two Qwen3 models. Effects and 95% problem-instance bootstrap confidence intervals are in percentage points. Yellow cells and ∗ mark statistically significant interactions. Estimates use unrounded rates.

		Coordinator- selected speaker	Broadcast query	Round robin	Current-speaker selection	Simultaneous responses
PIUs only	One PIU per participant	69.7%	59.8%	73.5%	75.3%	60.9%
	All-but-one PIU concentration	67.9%	60.9%	78.0%	78.5%	69.7%
PIUs and source material	One PIU per participant	35.9%	41.2%	39.6%	46.0%	28.5%
	All-but-one PIU concentration	35.6%	28.0%	27.8%	37.1%	23.2%

Appendix

Table 41: SARA episode success with exact PIUs and added source material. Rates pool three models and three repetitions over 44 problem instances. Both evidence conditions supply the same exact assigned PIUs. The PIUs-only rows are Multi-Agent PIU-Only; the other rows additionally supply the original private source material.

Communication structure	PIUs only	PIUs and source material	Interaction
Coordinator-selected speaker	$-1.8\;[-8.6,4.8]$	$-0.3\;[-6.1,5.6]$	$+1.5\;[-7.3,10.1]$
Broadcast query	$+1.0\;[-5.3,7.3]$	$-13.1\;[-19.7,-6.1]$	$-14.1\;[-22.7,-5.6]$
Round robin	$+4.5\;[-2.3,11.1]$	$-11.9\;[-17.9,-5.5]$	$-16.4\;[-25.3,-7.6]$
Current-speaker selection	$+3.3\;[-3.0,9.8]$	$-8.8\;[-15.2,-2.3]$	$-12.1\;[-19.4,-4.8]$
Simultaneous responses	$+8.8\;[2.8,15.2]$	$-5.3\;[-12.1,1.5]$	$-14.1\;[-23.0,-5.1]$

Appendix

Table 42: SARA allocation effects and source-material interactions. Allocation effects are All-but-one PIU concentration minus One PIU per participant. The interaction is the added-source allocation effect minus the Multi-Agent PIU-Only allocation effect. The table heading “PIUs only” denotes Multi-Agent PIU-Only. Entries are percentage points with 95% problem-instance bootstrap confidence intervals.

Communication structure	None	Assigned PIUs only	Assigned no PIU only	Both groups
Coordinator-selected speaker	67.9%	53.3%	38.1%	35.6%
Broadcast query	60.9%	47.2%	28.0%	28.0%
Simultaneous responses	69.7%	54.8%	22.5%	23.2%

Appendix

Table 43: SARA source-material recipients under All-but-one PIU concentration. All participants assigned PIUs receive their exact PIUs. Column headings specify which participants additionally receive private source material. Entries are episode success rates.

Source material for participants assigned no PIU	Complete transcript	Messages from participants assigned PIUs only	Effect	95% CI
Absent	69.9%	68.9%	$-1.0$	$[-5.3,3.3]$
Present	21.2%	68.9%	$+47.7$	$[38.9,56.3]$
Interaction			$+48.7$	$[38.9,58.6]$

Appendix

Table 44: SARA fixed-message intervention at the final-decision stage. Participants assigned PIUs receive no added source material in either row. Only the source-material access of participants assigned no PIU varies. The effect is episode success with their messages removed minus success with the complete transcript. The interaction is this removal effect with source material minus the same effect without it. Effects and 95% problem-instance bootstrap intervals are in percentage points.

Private-note tokens	Message tokens	GPT-5.5 CUAD	GPT-5.5 Multi-LexSum	Qwen3-32B CUAD	Qwen3-32B Multi-LexSum
64	32	70.8 $\pm$ 9.3	24.1 $\pm$ 7.9	79.2 $\pm$ 8.3	3.4 $\pm$ 3.4
64	64	70.8 $\pm$ 9.3	65.5 $\pm$ 8.8	75.0 $\pm$ 8.8	0.0 $\pm$ 0.0
64	128	70.8 $\pm$ 9.3	72.4 $\pm$ 8.3	91.7 $\pm$ 5.6	3.4 $\pm$ 3.4
64	256	70.8 $\pm$ 9.3	69.0 $\pm$ 8.6	83.3 $\pm$ 7.6	0.0 $\pm$ 0.0
128	32	79.2 $\pm$ 8.3	44.8 $\pm$ 9.2	75.0 $\pm$ 8.8	3.4 $\pm$ 3.4
128	64	79.2 $\pm$ 8.3	62.1 $\pm$ 9.0	75.0 $\pm$ 8.8	6.9 $\pm$ 4.7

Appendix

Table 46: Private-note and message token budgets by source corpus and model. Episode success rates (%); small

\pm

terms are binomial standard errors over 24 CUAD or 29 Multi-LexSum problem instances.

Local-transcript tokens	FIFO	Change from baseline	Random contiguous window	Change from baseline
16	82.8 $\pm$ 7.0	-3.4	75.9 $\pm$ 3.4	-10.3
32	82.8 $\pm$ 7.0	-3.4	83.9 $\pm$ 2.0	-2.3
64	86.2 $\pm$ 6.4	+0.0	81.6 $\pm$ 5.3	-4.6
128	79.3 $\pm$ 7.5	-6.9	85.1 $\pm$ 8.0	-1.1
256	86.2 $\pm$ 6.4	+0.0	82.8 $\pm$ 6.9	-3.4
Unlimited	86.2 $\pm$ 6.4	0.0	—	—

Appendix

Table 47: Local-transcript token budgets. Episode success rates (%) and changes from the unlimited FIFO baseline. Small

\pm

terms are binomial standard errors for FIFO and standard deviations over three seeds for random contiguous-window reading.

Use	CUAD-derived	MAUD-derived	Total
Training	334	0	334
Validation	36	0	36
MAUD-derived evaluation	0	26	26

Appendix

Table 48: Free-form legal problem instances. Only CUAD-derived training problem instances supply gradients. CUAD-derived validation contracts are absent from training, and MAUD-derived problem instances are opened only after CUAD-derived checkpoint selection.

Setting	Participant-only stage	Coordinator-only continuation	Joint two-policy training
Updated policy	Participant policy	Coordinator policy	Participant and coordinator policies
Fixed policy	Base coordinator policy	Trained participant policy	Neither

Appendix

Table 49: Training configurations under Coordinator-selected speaker. The participant-only stage and coordinator-only continuation form one two-stage policy-training experiment. Joint two-policy training begins independently from the base policy for both trainable policies.

Training stage	LLM-judged score
Base policies	12.14
Participant-only stage	16.84
Coordinator-only continuation	23.33

Appendix

Table 50: CUAD-derived validation results during two-stage policy training. Values are percentages.

Case	Base participant policy	Trained participant policy	Observed change
Liability limits	Both participant messages deny that the agreement contains liability limits; the final decision receives 0.00.	One participant message includes the mutual exclusion of lost profits and indirect, special, consequential, and punitive damages, including the product context and even-if-advised condition; the final decision receives 0.33.	Participant message includes information needed for a correct final decision
Compliance audit and access	Participant messages leave the right holder unspecified and substitute unrelated consumer-credit services; the final decision receives 0.00.	The two participant messages separately include Equidata’s at-any-time site-inspection right and its compliance audit at Equidata’s expense; the final decision receives 0.33 but still merges the two rights.	Different PIUs appear across participant messages
Insurance obligations	The first participant message includes the $1M coverage, but the second coordinator query repeats participant A and obtains an unrelated fiber-rights clause; the final decision receives 0.40.	After the trained participant policy produces a different first message, the fixed coordinator observes the changed transcript and queries participant B, which supplies the cancellation, non-renewal, modification, and 30-day notice condition; the final decision receives 0.80.	Changed participant message leads to a different next coordinator query

Appendix

Table 51: Illustrative changes after participant-only training. Only the participant policy changes. Values are LLM-judged scores; the examples are not frequency estimates and do not enter training or checkpoint selection.

Case	Base coordinator policy	Trained coordinator policy	Observed change
Distributor insurance	Participant messages include unrelated regulatory and committee-cost facts; the final decision receives 0.00.	Changed coordinator queries cause participant messages to include both the certificate-on-request clause and the reciprocal insurance obligation; the final decision receives 1.00.	Changed coordinator queries
Co-brand minimum commitment	Both participant messages contain the 5,000-unit initial order and the 10,000-unit/32-month guarantee, but the final decision confuses their roles; the final decision receives 0.33.	With substantively unchanged participant messages, the final decision correctly separates the initial order from the longer-term guarantee and receives 1.00.	Changed final decision from the same participant messages

Appendix

Table 52: Illustrative changes after the coordinator-only continuation. Only the coordinator policy changes; the trained participant policy remains fixed. Values are LLM-judged scores; the examples are not frequency estimates and do not enter training or checkpoint selection.

	FinDVer validation			FinLongDocQA test
Measurement	Base	Trained	$\Delta$	Base	Trained	$\Delta$
Same final-decision policy: LLM-judged score	34.10	46.72	+12.62	49.60	44.82	$-4.78$
Public messages	7.28	12.46	+5.18	9.30	13.43	+4.12
PIUs appearing in the communication transcript (%)	38.0	53.9	+15.9	63.1	63.1	0.0
PIUs appearing in the communication transcript with no incompatible message (%)	23.3	26.3	+3.0	55.6	58.8	+3.1
Answer components fully supported (%)	23.5	33.0	+9.5	26.2	25.0	$-1.2$

Appendix

Table 53: Communication-trajectory analysis. Values are means across problem instances. Percentages are shown except for public messages. The same 39 FinDVer validation and 40 held-out FinLongDocQA problem instances are paired across policies.

Multi-Agent Collaboration

Multi-agent collaboration studies how autonomous agents coordinate, share information, and combine capabilities to solve tasks together.

See all 312 papers

May 8, 2026

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls. Without enforcement, a team pass rate can mask whether agents actually coordinated or whether one role effectively did another role's work. We present TeamBench, a benchmark with 851 task templates and 931 seeded instances for evaluating agent coordination under operating system-enforced role separation. TeamBench separates specification access, workspace editing, and final certification across Planner, Executor, and Verifier roles, so that no role can read the full requirements, modify the workspace, and certify the final answer. Prompt-only and sandbox-enforced teams reach statistically indistinguishable pass rates, but prompt-only runs produce 3.6 times more cases where the verifier attempts to edit the executor's code. Verifiers approve 49% of submissions that fail the deterministic grader, and removing the verifier improves mean partial score in the ablation. Team value is also conditional. Teams benefit when single agents struggle, but hurt when single agents already perform well. A 40-session human study under the same role separation shows that our benchmark exposes interaction patterns that pass rate misses. Solo participants work through the task directly, human participants paired with agents often collapse into quick approval, and human teams spend more effort coordinating missing information across roles.

Yubin Kim, Chanwoo Park, Taehan Kim +9

MIT · Google Research · Independent Researcher +1

Multi-Agent Coordination LLM Agent Evaluation AI Agent Security AI Agent Benchmarks Task Decomposition Tool Access Control for LLM Agents+1

Ask PDF

Aug 4, 2026

Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems

Clinical decision support is moving toward committees of language-model agents deliberating on a shared workspace. We ask whether such committees can be gamed by shortcuts, cues a benchmark rewards but a clinician would ignore. Across seven cohorts on six public datasets spanning text (MedQA-USMLE, MedMCQA, MIMIC-CXR reports), imaging (NIH ChestX-ray14, MIMIC-CXR-JPG, CheXpert) and tabular ICU records (SUPPORT2), Gemini committees resist these cues in isolation (flip 5-16%), yet a socially plausible shortcut spreads: when two peers assert the same wrong answer, the holdout under test adopts it in 38% of cases, as does a false "pre-screen" system flag, on both capability tiers. Of three oversight agents, a gate cannot separate adoption from honest agreement (false-positive rate 100%); a same-lineage judge reading only the transcript flags adoption on text (precision 100%, recall 93%) but collapses onto the gate in imaging; a referee that privately re-queries the holdout transfers to imaging (77-88% precision, 13-21% false-positive rate). Tripling a cue's visual salience does not move contagion, whereas a second peer voice raises it by half again. Gaming a hidden rubric is near-silent: only 1/10 text and 1/134 imaging drifters name the rubric they moved toward. What games a committee is social plausibility, and only a referee independent of self-report catches it. Code: https://github.com/criticaldata/benchmaxxing

Sebastián Andrés Cajas Ordóñez, Agastya Munnangi, Aldo Marzullo +13

MIT Critical Data, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States · Georgia State University, Atlanta, Georgia, United States · Politecnico di Milano, Milan, Italy +12

Emergent Behavior in Multi-Agent Systems Multi-Agent LLM Systems AI Agent Safety Benchmarks AI Agent Reliability Clinical Decision Support LLM Agent Evaluation+3

Ask PDF

Sep 30, 2026

Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams

People are increasingly delegating tasks to AI agents, and those agents are increasingly encountering other people's agents over shared resources such as a codebase, a calendar, or a budget. When each agent acts for a different user with different goals, coordination often fails, and the group ends up worse off than if a single agent had acted for everyone. We study this multi-user, multi-agent setting across five frontier models and 77 scenarios in four environments: an API key environment in which agents share a compute budget, a clinic in which they share a calendar, a personal assistant environment in which they share a group order or booking, and a merge queue in which they share a release cutoff. In each scenario, we compare a single agent that serves every user (a coordinator) to a team in which each agent serves one user, with and without a communication channel between the agents. Teams deliver worse group outcomes than the coordinator in every environment: without a channel, they completely collapse in two environments, and even with one, coordination overhead creates substantial gaps. For example, in the personal assistant environment, the coordinator fulfills a targeted user request about twice as often as teams. We identify distinct behaviors associated with this poor group-level performance, including stalling as teams grow, overriding each other's actions, and fabricating claims. We find effective but environment-specific mitigations, such as a team lead, explicit procedural instructions, and a platform check that makes an agent read its peers' messages before committing. We will release the API key, clinic, and personal assistant environments as MAMUBench, comprising 74 scenarios for evaluating multi-user, multi-agent coordination.

Sahan Paliskara, Nattaput Namchittai, Andrew Lampinen

Independent Researcher · Anthropic

Multi-Agent LLM Systems Multi-Agent Coordination AI Agent Benchmarks Multi-Agent Systems Multi-Agent Communication

Ask PDF

Abstract

Multi-Agent Collaboration

Explore similar work

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems

Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams