Belief-Conditioned Multi-Agent Actor Critic for Decentralized LLM Collaboration

Tianle Chen; Shuo Liu; Christopher Amato

cs.AIOct 26, 2026

Belief-Conditioned Multi-Agent Actor Critic for Decentralized LLM Collaboration

Authors: Tianle Chen, Shuo Liu, Christopher Amato

Organizations: Khoury College of Computer Sciences, Northeastern University, Boston, MA, 02215, USA

Abstract

Recent work has explored optimizing LLM collaboration through Multi-Agent Reinforcement Learning (MARL), but many approaches still rely on predefined workflows, limiting efficiency and scalability. Decentralized LLM collaboration is challenging because agents execute independently with limited communication, share a joint objective, and must infer useful information from communication traces. In this paper, we propose BCMAAC, a Belief-Conditioned Multi-Agent Actor-Critic method that maintains beliefs over contextual information and uses them to facilitate training. Experiments on Minecraft game-playing, web searching and tool-using tasks show that BCMAAC improves task performance and collaborative behavior compared to naive actor-critic baselines. The improvements are particularly pronounced in multi-turn tasks with richer contextual information, where belief modeling provides a clear advantage.

Figures & tables

Figure 1: Illustration of the BCMAAC framework.

	Minecraft		Web		Tool
Method	Resource	Bridge	Cross-Lingual	Multi-Source	Stateful	Filesystem
	Gathering	Building	Gathering	Gathering	Tool	Wrangling
	Success (%)	Success (%)	Match Score (%)	Match Score (%)	Success (%)	Success (%)
Raw Model	$0.0_{\pm 0.0}$	$0.0_{\pm 0.0}$	$45.0_{\pm 0.0}$	$11.3_{\pm 0.0}$	$33.3_{\pm 0.0}$	$53.8_{\pm 0.0}$
AC	$70.1_{\pm 5.7}$	$30.1_{\pm 7.2}$	$51.9_{\pm 3.4}$	$19.7_{\pm 3.8}$	$55.8_{\pm 7.5}$	$66.9_{\pm 8.0}$
Parallel	$0.0_{\pm 0.0}$	$0.0_{\pm 0.0}$	$37.3_{\pm 1.0}$	$16.9_{\pm 2.7}$	$33.0_{\pm 5.6}$	$25.3_{\pm 0.0}$

Table 1: Evaluation results across Minecraft, Web, and Tool collaboration. AC denotes single-agent actor-critic method; Parallel and Pipeline represents 2 test-time prompt-based multi-agent collaboration methods; MAAC is the centralized-critic multi-agent actor-critic baseline; BCMAAC-P and BCMAAC-B are variants that only partner-action and task-belief auxiliary beliefs, respectively. Success is the success rate and match score is the normalized answer-matching score, both are reported as percentages and averaged over evaluation runs. Standard deviations are reported in subscripts.

Panel of Figure 2 — Figure 2: Minecraft tasks. (a) In Bridge Building , agents start at the blue beacon endpoints and must construct a white-block bridge under limited vision. Brown wooden pillars are safe, whereas gray stone pillars with wolves are dangerous decoys. (b) In Resource Gathering , agents with complementary tools observe only local regions and collect from trees, stone, and minerals of iron.

Panel of Figure 3 — Figure 3: A comparison between BCMAAC (red) and MAAC (blue) across the six tasks. The y-axis denotes the expected return, and the x-axis denotes the environment steps. Results are all averaged over five runs, and shaded regions indicate the standard deviation. Curves are smoothed using a time-weighted exponential moving average.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Split	Horizon	Metric
Resource Gathering	$7/7$	$4$	success
Bridge Building	$6/3$	$4$	success
Cross-Lingual	$120/30$	$2$	match score
Multi-Source	$30/15$	$3$	match score
Stateful Tool	$25/15$	$4$	success
Filesystem Wrangling	$22/13$	$4$	success

Appendix

Table 2: Dataset splits and primary metrics.

Task	Training hyperparameters
Resource Gathering	$180$ epochs; rollout/batch $8/4$ ; $\gamma=0.90$ ; actor/critic LR $2.5{\times}10^{-6}$ ; context LR $3.0{\times}10^{-5}$ ; max tokens $160$ .
Bridge Building	$180$ epochs; rollout/batch $8/4$ ; $\gamma=0.90$ ; actor/critic LR $2.5{\times}10^{-6}$ ; context LR $3.0{\times}10^{-5}$ ; max tokens $160$ .
Cross-Lingual	$30$ epochs; rollout/batch $16/4$ ; $\gamma=0.95$ ; actor/critic LR $2.0{\times}10^{-6}$ ; context LR $1.0{\times}10^{-5}$ ; max tokens $192$ .
Multi-Source	$30$ epochs; rollout/batch $12/2$ or $16/4$ ; $\gamma=0.95$ ; actor/critic LR $1.0{\times}10^{-6}$ ; context LR $1.0{\times}10^{-4}$ ; max tokens $192$ .
Stateful Tool	$120$ epochs; rollout/batch $10/2$ ; actor/critic LR $1.5{\times}10^{-6}$ ; context LR $5.0{\times}10^{-5}$ ; max tokens $160$ .
Filesystem Wrangling	$120$ epochs; rollout/batch $10/2$ ; actor/critic LR $1.5{\times}10^{-6}$ ; context LR $3.0{\times}10^{-5}$ ; max tokens $160$ .

Appendix

Table 3: Main training hyperparameters. The context learning rate applies to BCMAAC variants.

Method	Execution	Training signal	Notes
Raw Model	Decentralized LLM agents without RL updates.	None.	Uses the same prompts and parsers as the trained methods; measures the capability of the initial instruction model.
AC	Single-agent actor-critic.	Return regression and policy-gradient updates from the task reward.	Tests how far task-specific RL improves a single actor without learned multi-agent coordination.
Parallel	Prompt-only multi-agent parallel execution.	None.	Agents act in parallel from their own prompts and received messages; no policy update or centralized critic.
Pipeline	Prompt-only serial execution.	None.	Agents act in a fixed order, so later agents can condition on earlier outputs; the collaboration pattern is manually specified.
MAAC	Decentralized actors with centralized training.	Actor-critic updates with a centralized critic over joint histories.	The closest RL baseline; actors execute from local histories while the critic estimates team return during training.
BCMAAC-P	BCMAAC without task-belief supervision.	Actor-critic updates plus partner-action / action-preference auxiliary supervision.	Isolates the effect of modeling teammate behavior / action preferences.

Appendix

Table 4: Detailed definitions of baselines and ablation variants.

Task	Structured grid	Scalars	Task-belief target
Resource Gathering	$10$ spatial channels over the map.	$11$ normalized scalar features.	Non-empty resource cells relevant to the agent’s tool role.
Bridge Building	$12$ spatial channels over the map.	$8$ normalized scalar features.	True support pillars among candidate pillars.
Cross-Lingual Gathering	$8$ piece channels over the evidence list.	$20$ normalized scalar features.	Evidence pieces in the annotated support chain.
Multi-Source Gathering	$8$ piece channels over table and passage pieces.	$20$ normalized scalar features.	Evidence pieces in the annotated cross-source support chain.
Stateful Tool	$8$ channels over $6$ belief slots.	$16$ normalized scalars.	Required progress slots (whether the executor holds the key fact, whether the setting is ready, whether the target action is done, etc.).
Filesystem Wrangling	$8$ channels over $6$ belief slots.	$16$ normalized scalars.	Required progress slots (whether the target is located, whether the executor knows the path, whether the destination is ready, whether the target action is done, etc.).

Appendix

Table 5: Structured side information and task-belief targets used by BCMAAC.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,032 papers

Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards
Oct 13, 2026Keru Chen, Sen Lin, Yingbin Liang +2LLM Agent Training Multi-Agent Collaboration
School of ECEE, Arizona State University · Computer Science Department, University of Houston · Department of ECE, The Ohio State University +1
Improving LLM Collaboration via Multi-Agent Preference Learning
Sep 26, 2026Shuo Liu, Xinzichen Li, Tianle Chen +1Multi-Agent Collaboration RL for LLM Agents
Northeastern University, Boston
TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination
May 1, 2026Yi Xie, Siao Liu, Falong Fan +3Multi-Agent System Optimization Multi-Agent Coordination
Department of Electrical & Computer Engineering, University of Arizona · Future Science and Engineering College, Soochow University · INSAIT, Sofia University ”St. Kliment Ohridski” +1

Oct 13, 2026J/K move · Enter open · S save

Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards

Keru Chen, Sen Lin, Yingbin Liang, Nathaniel D. Bastian+1

School of ECEE, Arizona State University · Computer Science Department, University of Houston · Department of ECE, The Ohio State University+1

LLM Agent Training Multi-Agent Collaboration Multi-LLM Collaboration Reinforcement Learning with Verifiable Rewards Reward Shaping

Open paper AskPDF arXiv

Effective collaboration among large language model (LLM) agents in multi-agent systems requires sharing relevant evidence, requesting missing information, and incorporating teammates' contributions into decisions. Training these behaviors is challenging because final task outcomes provide limited feedback on the value of individual messages. We introduce CoSIR (Collaboration via Surprisal-based Information Rewards), a post-training approach that uses a simple, verifiable environment to teach collaboration behaviors that generalize to more complex, unseen tasks. We propose a Bayesian evaluator that scores shared information by its reduction in team surprisal, defined as the sum of the negative log probabilities assigned to the correct option given each agent's available evidence. Requests are scored by the potential reduction in the same quantity if the requested information were truthfully provided. Separate rewards encourage agents to use the received information in their private choices and calculations. We post-train Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it through self-play and evaluate transfer to seven unseen natural-language collaboration tasks. CoSIR achieves the highest aggregate score, while its performance in general tasks remains competitive. Behavioral analyses show increased disclosure of private evidence and fewer judged failures to use teammates' input.

Abstract

Multi-Agent LLM Systems

Explore similar work

Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards

Improving LLM Collaboration via Multi-Agent Preference Learning

TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination