Recent work has explored optimizing LLM collaboration through Multi-Agent Reinforcement Learning (MARL), but many approaches still rely on predefined workflows, limiting efficiency and scalability. Decentralized LLM collaboration is challenging because agents execute independently with limited communication, share a joint objective, and must infer useful information from communication traces. In this paper, we propose BCMAAC, a Belief-Conditioned Multi-Agent Actor-Critic method that maintains beliefs over contextual information and uses them to facilitate training. Experiments on Minecraft game-playing, web searching and tool-using tasks show that BCMAAC improves task performance and collaborative behavior compared to naive actor-critic baselines. The improvements are particularly pronounced in multi-turn tasks with richer contextual information, where belief modeling provides a clear advantage.
Figures & tables
Figure 1: Illustration of the BCMAAC framework.
Minecraft
Web
Tool
Method
Resource
Bridge
Cross-Lingual
Multi-Source
Stateful
Filesystem
Gathering
Building
Gathering
Gathering
Tool
Wrangling
Success (%)
Success (%)
Match Score (%)
Match Score (%)
Success (%)
Success (%)
Raw Model
0.0±0.0
0.0±0.0
45.0±0.0
11.3±0.0
33.3±0.0
53.8±0.0
AC
70.1±5.7
30.1±7.2
51.9±3.4
19.7±3.8
55.8±7.5
66.9±8.0
Parallel
0.0±0.0
0.0±0.0
37.3±1.0
16.9±2.7
33.0±5.6
25.3±0.0
Table 1: Evaluation results across Minecraft, Web, and Tool collaboration. AC denotes single-agent actor-critic method; Parallel and Pipeline represents 2 test-time prompt-based multi-agent collaboration methods; MAAC is the centralized-critic multi-agent actor-critic baseline; BCMAAC-P and BCMAAC-B are variants that only partner-action and task-belief auxiliary beliefs, respectively. Success is the success rate and match score is the normalized answer-matching score, both are reported as percentages and averaged over evaluation runs. Standard deviations are reported in subscripts.
Figure 2: Minecraft tasks. (a) In Bridge Building , agents start at the blue beacon endpoints and must construct a white-block bridge under limited vision. Brown wooden pillars are safe, whereas gray stone pillars with wolves are dangerous decoys. (b) In Resource Gathering , agents with complementary tools observe only local regions and collect from trees, stone, and minerals of iron.
Figure 3: A comparison between BCMAAC (red) and MAAC (blue) across the six tasks. The y-axis denotes the expected return, and the x-axis denotes the environment steps. Results are all averaged over five runs, and shaded regions indicate the standard deviation. Curves are smoothed using a time-weighted exponential moving average.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Split
Horizon
Metric
Resource Gathering
7/7
4
success
Bridge Building
6/3
4
success
Cross-Lingual
120/30
2
match score
Multi-Source
30/15
3
match score
Stateful Tool
25/15
4
success
Filesystem Wrangling
22/13
4
success
Appendix
Table 2: Dataset splits and primary metrics.
Task
Training hyperparameters
Resource Gathering
180 epochs; rollout/batch 8/4 ; γ=0.90 ; actor/critic LR 2.5×10−6 ; context LR 3.0×10−5 ; max tokens 160 .
Bridge Building
180 epochs; rollout/batch 8/4 ; γ=0.90 ; actor/critic LR 2.5×10−6 ; context LR 3.0×10−5 ; max tokens 160 .
Cross-Lingual
30 epochs; rollout/batch 16/4 ; γ=0.95 ; actor/critic LR 2.0×10−6 ; context LR 1.0×10−5 ; max tokens 192 .
Multi-Source
30 epochs; rollout/batch 12/2 or 16/4 ; γ=0.95 ; actor/critic LR 1.0×10−6 ; context LR 1.0×10−4 ; max tokens 192 .
Stateful Tool
120 epochs; rollout/batch 10/2 ; actor/critic LR 1.5×10−6 ; context LR 5.0×10−5 ; max tokens 160 .
Filesystem Wrangling
120 epochs; rollout/batch 10/2 ; actor/critic LR 1.5×10−6 ; context LR 3.0×10−5 ; max tokens 160 .
Appendix
Table 3: Main training hyperparameters. The context learning rate applies to BCMAAC variants.
Method
Execution
Training signal
Notes
Raw Model
Decentralized LLM agents without RL updates.
None.
Uses the same prompts and parsers as the trained methods; measures the capability of the initial instruction model.
AC
Single-agent actor-critic.
Return regression and policy-gradient updates from the task reward.
Tests how far task-specific RL improves a single actor without learned multi-agent coordination.
Parallel
Prompt-only multi-agent parallel execution.
None.
Agents act in parallel from their own prompts and received messages; no policy update or centralized critic.
Pipeline
Prompt-only serial execution.
None.
Agents act in a fixed order, so later agents can condition on earlier outputs; the collaboration pattern is manually specified.
MAAC
Decentralized actors with centralized training.
Actor-critic updates with a centralized critic over joint histories.
The closest RL baseline; actors execute from local histories while the critic estimates team return during training.
BCMAAC-P
BCMAAC without task-belief supervision.
Actor-critic updates plus partner-action / action-preference auxiliary supervision.
Isolates the effect of modeling teammate behavior / action preferences.
Appendix
Table 4: Detailed definitions of baselines and ablation variants.
Task
Structured grid
Scalars
Task-belief target
Resource Gathering
10 spatial channels over the map.
11 normalized scalar features.
Non-empty resource cells relevant to the agent’s tool role.
Bridge Building
12 spatial channels over the map.
8 normalized scalar features.
True support pillars among candidate pillars.
Cross-Lingual Gathering
8 piece channels over the evidence list.
20 normalized scalar features.
Evidence pieces in the annotated support chain.
Multi-Source Gathering
8 piece channels over table and passage pieces.
20 normalized scalar features.
Evidence pieces in the annotated cross-source support chain.
Stateful Tool
8 channels over 6 belief slots.
16 normalized scalars.
Required progress slots (whether the executor holds the key fact, whether the setting is ready, whether the target action is done, etc.).
Filesystem Wrangling
8 channels over 6 belief slots.
16 normalized scalars.
Required progress slots (whether the target is located, whether the executor knows the path, whether the destination is ready, whether the target action is done, etc.).
Appendix
Table 5: Structured side information and task-belief targets used by BCMAAC.
Effective collaboration among large language model (LLM) agents in multi-agent systems requires sharing relevant evidence, requesting missing information, and incorporating teammates' contributions into decisions. Training these behaviors is challenging because final task outcomes provide limited feedback on the value of individual messages. We introduce CoSIR (Collaboration via Surprisal-based Information Rewards), a post-training approach that uses a simple, verifiable environment to teach collaboration behaviors that generalize to more complex, unseen tasks. We propose a Bayesian evaluator that scores shared information by its reduction in team surprisal, defined as the sum of the negative log probabilities assigned to the correct option given each agent's available evidence. Requests are scored by the potential reduction in the same quantity if the requested information were truthfully provided. Separate rewards encourage agents to use the received information in their private choices and calculations. We post-train Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it through self-play and evaluate transfer to seven unseen natural-language collaboration tasks. CoSIR achieves the highest aggregate score, while its performance in general tasks remains competitive. Behavioral analyses show increased disclosure of private evidence and fewer judged failures to use teammates' input.
Keru Chen, Sen Lin, Yingbin Liang +2
School of ECEE, Arizona State University · Computer Science Department, University of Houston · Department of ECE, The Ohio State University +1
Several works have explored multi-agent reinforcement learning (MARL) in LLM collaboration. However, constructing reliable rewards is difficult in practice, as complete and accurate metrics are often unavailable and hard to aggregate. Preference learning provides an alternative by learning from comparative human or AI feedback. Yet, its extension to multi-agent systems remains underexplored. To address this gap, we formulate preference-based multi-agent systems (MAS) from decentralized and centralized collaboration perspectives. We also introduce a general multi-agent preference learning framework (MAPL) to solve these problems. MAPL allows iterative updates by comparing the current solution with decentralized or centralized solutions generated by various agents. We instantiate MAPL using MARL from human feedback (MARLHF) with a learned reward model and multi-agent direct preference optimization (MADPO). Experiments on collaborative writing, coding, tool use, and travel planning show that MAPL can improve collaboration quality and efficiency while approaching the performance of MARL with fixed, well-defined rewards. Within MAPL, MARLHF generally outperforms MADPO on most tasks but remains sensitive to data coverage, agent and comparator models, and the underlying MARL algorithms.
Multi-agent LLM systems have shown promise for complex reasoning, yet recent evaluations reveal they often underperform single-model baselines. We identify a structural failure mode in sequential fine-tuning of shared-context teams: updating one agent shifts the team's context distribution, and when subsequent updates are evaluated on cached rollouts, this mismatch compounds. We formalize this as the compounding occupancy shift and prove that stale-occupancy evaluation incurs a penalty that scales quadratically with the number of agents. In contrast, intermediate-occupancy evaluation reduces this to linear scaling. We propose TeamTR, a trust-region framework that resamples trajectories after each component update and enforces per-agent divergence control, yielding rigorous per-update and per-stage improvement lower bounds. Experiments show that TeamTR outperforms single-agent and sequential baselines with 7.1% on average, mitigates coordination regressions, and supports plug-and-play component replacement. Code is available at https://github.com/Yydc/TeamTR.
Yi Xie, Siao Liu, Falong Fan +3
Department of Electrical & Computer Engineering, University of Arizona · Future Science and Engineering College, Soochow University · INSAIT, Sofia University ”St. Kliment Ohridski” +1