Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards

Keru Chen; Sen Lin; Yingbin Liang; Nathaniel D. Bastian; Shaofeng Zou

cs.AIOct 13, 2026

Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards

Authors: Keru Chen, Sen Lin, Yingbin Liang, Nathaniel D. Bastian, Shaofeng Zou

Organizations: School of ECEE, Arizona State University · Computer Science Department, University of Houston · Department of ECE, The Ohio State University · Whiting School of Engineering, Johns Hopkins University

Abstract

Effective collaboration among large language model (LLM) agents in multi-agent systems requires sharing relevant evidence, requesting missing information, and incorporating teammates' contributions into decisions. Training these behaviors is challenging because final task outcomes provide limited feedback on the value of individual messages. We introduce CoSIR (Collaboration via Surprisal-based Information Rewards), a post-training approach that uses a simple, verifiable environment to teach collaboration behaviors that generalize to more complex, unseen tasks. We propose a Bayesian evaluator that scores shared information by its reduction in team surprisal, defined as the sum of the negative log probabilities assigned to the correct option given each agent's available evidence. Requests are scored by the potential reduction in the same quantity if the requested information were truthfully provided. Separate rewards encourage agents to use the received information in their private choices and calculations. We post-train Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it through self-play and evaluate transfer to seven unseen natural-language collaboration tasks. CoSIR achieves the highest aggregate score, while its performance in general tasks remains competitive. Behavioral analyses show increased disclosure of private evidence and fewer judged failures to use teammates' input.

Figures & tables

Figure 1: Overview of CoSIR. Agents receive different parts of a rating table and exchange messages before choosing an option. A Bayesian evaluator scores shared ratings by how they change the probability of the correct option and scores requests by the potential value of a truthful answer. Private-answer rewards assess option choices and calculations using the ratings available to each agent. Self-play updates a policy shared by all agents, which is then evaluated on seven unseen collaboration tasks. The table and briefings illustrate one training instance.

	Unseen collaboration tasks: mean $\pm$ std							Overall transfer
Method	Hidden Bench	SILO	SplitQA	MathChat	Bench Form	Einstein	Collab. Tax	SUM	$\Delta$ SUM
Qwen3-8B
Base	$64.23_{\pm 3.85}$	$63.53_{\pm 3.56}$	$57.50_{\pm 6.32}$	$78.48_{\pm 1.97}$	$49.80_{\pm 0.99}$	$3.58_{\pm 0.69}$	$35.48_{\pm 7.44}$	352.60	0.00
ToM	$68.85_{\pm 4.24}$	$\mathbf{71.70}_{\pm 3.99}$	$56.43_{\pm 1.84}$	$73.95_{\pm 0.20}$	$51.05_{\pm 3.89}$	$5.42_{\pm 0.17}$	$26.78_{\pm 2.92}$	354.17	+1.57
TalkHier	$65.38_{\pm 5.25}$	$68.63_{\pm 1.60}$	$53.93_{\pm 0.71}$	$81.96_{\pm 2.70}$	$\mathbf{56.80}_{\pm 5.37}$	$2.50_{\pm 0.33}$	$35.45_{\pm 1.14}$	364.65	+12.06
Outcome-RL	$67.31_{\pm 4.76}$	$68.25_{\pm 3.54}$	$57.50_{\pm 2.95}$	$79.55_{\pm 3.61}$	$51.80_{\pm 1.98}$	$4.33_{\pm 1.09}$	$34.98_{\pm 3.92}$	363.72	+11.12

Table 1: Transfer to seven unseen collaboration tasks. Task scores (%) show mean

\pm

standard deviation. SUM is the sum of seven task scores, each measured on a 0–100 scale.

\Delta

SUM is the change from Base for the same model. Shading marks CoSIR. Bold marks the best score within each model, including ties. SILO denotes SILO-BENCH and Collab. Tax denotes Collaboration Tax.

Task	Behavior	Qwen3-8B	Qwen3-4B	Gemma-3-4B-it
HiddenBench	Private facts disclosed $\uparrow$	68.5 $\rightarrow$ 74.3	24.5 $\rightarrow$ 33.9	9.9 $\rightarrow$ 15.3
	Messages responding to teammates $\uparrow$	59.4 $\rightarrow$ 65.8	64.0 $\rightarrow$ 73.4	93.1 $\rightarrow$ 95.4
SILO-BENCH	Agents listing all own data $\uparrow$	55.3 $\rightarrow$ 65.4	74.8 $\rightarrow$ 78.4	76.8 $\rightarrow$ 85.9
	Result-only messages $\downarrow$	16.2 $\rightarrow$ 12.8	10.2 $\rightarrow$ 8.1	10.6 $\rightarrow$ 9.3
SplitQA	Holders stating supporting facts $\uparrow$	52.9 $\rightarrow$ 58.2	56.8 $\rightarrow$ 62.9	33.6 $\rightarrow$ 35.7
MathChat	Messages with correction language $\uparrow$	2.5 $\rightarrow$ 11.1	4.9 $\rightarrow$ 6.2	40.1 $\rightarrow$ 44.4

Table 2: Observed collaboration behaviors (%). Each pair reads Base

\rightarrow

CoSIR . Arrows beside the measures indicate the intended direction. Disclosure measures cover any discussion round. Correction language records attempts to correct an error, without judging whether the correction succeeds. Appendix J defines the denominators.

Failure type	Base	CoSIR
Specification and system design	88.2	88.3
Disobey task specification	5.6	6.1
Disobey role specification	0.0	0.0
Repeated steps	85.1	85.6
Loss of conversation history	9.5	9.8
Unaware of when to stop	18.5	22.4

Table 3: MAST failures (%) pooled across three models on HiddenBench, SplitQA, and MathChat ( Cemri et al., 2025 ) . Shaded rows give category totals, and bold marks the lower value. The final row gives the mean count of failure types. Appendix K defines the aggregation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Weight	Agents	Private ratings	Shared	Holders shown	Rounds
A	30%	4	Usually one cell per agent	3–5	No	3–6
B	15%	2	Usually two cells per agent	3–5	No	3–5
C	20%	4	Portions of sizes $(8,4,4,4)$	0	No	3–6
D	15%	2	$(6,6)$ , $(7,5)$ , or $(8,4)$ cells	4	Yes	4–8
E	20%	3	$(5,5,5)$ , $(7,5,3)$ , or $(6,6,2)$ cells	4	No	4–8

Appendix

Table 4: Information configurations. Parenthesized counts give the ratings allocated across agents. The number of discussion rounds is sampled uniformly within each listed range.

Setting	Value
Games per collection round	60
Policy messages per speaking state	16
Policy answers per answer state	8
Additional demonstrations per group	0–2
Policy sampling temperature	1.2
Generated message limit	1,024 tokens

Appendix

Table 5: Common sampling and optimization settings. A collection round means generating fresh games and then training on their candidate responses.

Backbone	Round 1	Round 2	Round 3
Qwen3-8B	$1\times 10^{-5}$	$1\times 10^{-5}$	$2\times 10^{-5}$
Qwen3-4B	$1\times 10^{-5}$	$1\times 10^{-5}$	$2\times 10^{-5}$
Gemma-3-4B-it	$1\times 10^{-5}$	$1\times 10^{-5}$	$2\times 10^{-5}$

Appendix

Table 6: Learning rates for the three collection rounds. Each rate is fixed within its round.

Task	Response stage	Temperature	Token limit	Thinking
HiddenBench	Discussion	0.7	320	Off
	Final answer	0	10,240	On
SILO-BENCH	Discussion	0.7	320	Off
	Final answer	0	6,144	On
SplitQA	Discussion	0.7	320	Off
	Final answer	0	6,144	On

Appendix

Table 7: Generation settings for the seven unseen collaboration tasks. The first three tasks use separate settings for discussion and final answers. SILO-BENCH permits one further answer attempt with thinking disabled when needed, as described in the text.

Model	Policy	GSM8K	MMLU	IFEval
Qwen3-8B	Base	93.9	77.6	84.4
	CoSIR	93.7	77.8	84.1
Qwen3-4B	Base	93.3	75.3	81.5
	CoSIR	93.1	74.7	81.8
Gemma-3-4B-it	Base	88.9	63.5	75.1
	CoSIR	89.7	63.8	75.0

Appendix

Table 8: General-capability scores (%) before and after collaboration training.

Explore similar work

Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents
Nov 11, 2025Chih-Hsuan, Yang, Tanwi Mallick +5Multi-Agent LLM Systems Cooperative Game Theory
Bella · Argonne National Laboratory · & University of Chicago
More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration
Apr 9, 2026Advait Yadav, Sid Black, Oliver SourbutMulti-Agent Coordination Multi-Agent Collaboration
MATS · University of Illinois Urbana-Champaign · UK AI Security Institute +1
Beyond Perfect Teammates: Multi-Agent Collaboration with Unreliable Colleagues
Oct 12, 2026Chen Huang, Yiyang Zhang, Yuhao Wu +2Multi-Agent Collaboration Multi-LLM Collaboration

Nov 11, 2025J/K move · Enter open · S save

Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents

Chih-Hsuan, Yang, Tanwi Mallick, Le Chen+4

Bella · Argonne National Laboratory · & University of Chicago

Multi-Agent LLM Systems Cooperative Game Theory Process Reward Models Credit Assignment in RL LLM Agent Training

Open paper AskPDF arXiv

Large Language Models (LLMs) in multi-agent systems (MAS) have shown promise for complex tasks, yet current training methods lack principled ways to connect system-level evaluation with agent- and message-level learning. We propose a theoretical framework that unifies cooperative game-theoretic attribution with process reward modeling to transform system evaluation to agent credit to response-level signals. Unlike prior approaches that rely only on attribution (Shapley) or step-level labels (PRM), our method produces local, signed, and credit-conserving signals. In success cases, Shapley-based credit assignment fairly allocates outcomes across agents and is refined into per-message rewards that promote cooperation while discouraging redundancy or sabotage; in failure cases, first-error localization yields repair-aware preferences that penalize harmful steps while rewarding corrective attempts. The resulting signals are bounded, cooperative, and directly compatible with reinforcement- or preference-based post-training, providing a unified and auditable pathway from global evaluation to local supervision in LLM multi-agent training. Our contribution is conceptual: we present a theoretical foundation and training signals, leaving empirical validation for future work.

Abstract

Explore similar work

Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents

More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration

Beyond Perfect Teammates: Multi-Agent Collaboration with Unreliable Colleagues