As agentic AI becomes more widespread, agents with distinct and possibly conflicting goals will interact in complex ways. These multi-agent interactions pose a fundamental challenge, particularly in social dilemmas, where agents' individual incentives can undermine collective welfare. While reinforcement learning (RL) has been effective for aligning large language models (LLMs) in the single-agent regime, prior small-network results suggest that standard RL in multi-agent settings often converges to defecting, self-interested policies. We show the same effect in LLMs: despite cooperative priors, RL-trained LLM agents develop opportunistic behavior that can exploit even advanced closed-source models. To address this tendency of RL to converge to poor equilibria, we adapt a recent opponent-learning awareness algorithm, Advantage Alignment, to fine-tune LLMs toward multi-agent cooperation and non-exploitability. We then introduce a group-relative baseline that simplifies advantage computation in iterated games, enabling multi-agent training at LLM scale. We also contribute a novel social dilemma environment, Trust-and-Split, which requires natural language communication to achieve high collective welfare. Across a wide range of social dilemmas, policies learned with Advantage Alignment achieve higher collective payoffs while remaining robust against exploitation by greedy agents. We release all of our code to support future work on multi-agent RL training for LLMs.
Figures & tables
Figure 1: Naive MARL consistently converges to suboptimal greedy behaviour in social dilemmas. Training curves of multi-agent GRPO on several open-source LLMs across IPD, Split No-Comm, and Trust-and-Split. In all environments, average rewards converge to the greedy payoff levels, showing that naive MARL drives LLMs toward defecting strategies in social dilemmas.
Figure 2: One round of the proposed Trust-and-Split Social Dilemma . Each player receives a private rock-paper-scissors hand that determines how much they value the coins, sends one message in turn, and then submits a proposal. Payoffs follow the split rule. Both hands and proposals are revealed before the next round starts.
Figure 3: Advantage Alignment learns to cooperate selectively while remaining robust to defection. Average rewards for an always-cooperate (Coop) agent, an always-defect (Defect) agent, an untrained model prompted to play tit-for-tat-like strategies (exact prompts are given in Appendix I ), and an Advantage Alignment (AdAlign) agent. In IPD (a) and Split No-Comm (b), AdAlign achieves near-cooperative payoffs with itself and Coop, while defecting against Defect. In Trust-and-Split (c), AdAlign cooperates with itself and with cooperative agents trained using multi-agent GRPO with summed rewards (GRPO-SR), while avoiding exploitation by agents trained with multi-agent GRPO. All learned agents are Qwen-2.5-7B-Instruct ( Team, 2024b ) trained with 8 random seeds. Learned agents are always paired by seed (seed i plays against seed i ). Each cell reports the mean per-round reward over 256 games per seed, aggregated as mean ± standard deviation across the 8 seeds.
Figure 4: Example Trust-and-Split interaction showing the tit-for-tat behaviour learned by Advantage Alignment. After Bob defects (as seen in the prompt summary of Alice for round 7), Alice defects in round 7, then returns to cooperation in round 9 once Bob cooperates again (shown in the prompt summary of Alice for round 9).
Figure 5: GPT-OSS-120B is exploitable by a learned agent, whereas our AdAlign is robust. (a) Training a multi-agent GRPO agent against a fixed gpt-oss-120b opponent in Trust-and-Split steadily increases the RL agent’s reward while reducing gpt-oss-120b ’s reward, indicating successful exploitation. Results in (a) are averaged over 3 seeds. (b) When training multi-agent GRPO against fixed AdAlign agents, the RL agent instead converges to cooperation, showing that the Advantage Alignment policy is robust to RL agents trained against it. Results in (b) are averaged over 8 seeds.
Method
IPD
Split No-Comm
Trust-and-Split
Full method
2.96±0.10
111.38±16.98
41.34±8.12
Without agent buffer
2.84±0.14
82.72±0.17
42.12±0.45
Without group baseline
2.96±0.10
108.27±17.30
26.96±0.96
Without AdAlign term ( β=0 )
1.0±0.0
83.45±2.77
29.04±0.92
Table 1: Ablations of Advantage Alignment training components
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
IPD
Split No-Comm
Trust-and-Split
Optimizer
Adam
Adam
Adam
Sampling Temperature
1.0
1.0
1.0
Learning Rate
3×10−6
3×10−6
3×10−6
Number of Rounds
10
10
10
Self-play Used
Yes
Yes
Yes
LoRA Rank
32
32
32
Appendix
Table 2: Hyperparameters for IPD, Split No-Comm, and Trust-and-Split experiments.
Hyperparameter
Value
Batch Size
64
Reward Normalization Constant
100.0
Entropy Coefficient
0.0
KL Coefficient
0.001
Discount Factor
0.96
gpt-oss-120b Reasoning Effort
Low
Appendix
Table 3: Hyperparameters for multi-agent GRPO against gpt-oss-120b .
Figure 7: GPT-5 nano is exploitable by a learned agent. Training a multi-agent GRPO agent against a fixed GPT-5 nano opponent in Trust-and-Split steadily increases the RL agent’s reward while reducing GPT-5 nano ’s reward, indicating successful exploitation.
Figure 8: Advantage alignment training curves (for three seeds) when setting β=0 . This shows that the agent replay buffer is not sufficient to learn cooperative strategies.
Figure 9: Average rewards during training for Advantage Alignment across our testbed environments with multiple random seeds. The method learns to maximize average reward in the majority of seeds (6 out of 8), demonstrating robust performance across environments. The corresponding results relating to robustness against exploitation are shown in Figures 3 b) and 3 c).
Figure 10: Average proposals of AdAlign agents in Trust-and-Split, conditioned on whether their hand has a higher or lower value in the rock–paper–scissors ordering. Agents propose higher amounts (converging toward 10) when holding the upper hand and lower amounts (converging toward 0) when holding the lower hand, a strategy that maximizes collective payoff. This pattern occurs in the majority of seeds (6 out of 8), demonstrating the robustness of the method.
Model
Provider
License / Terms
Qwen-2.5-7B-Instruct [ Team, 2024b ]
Alibaba / Qwen
Apache License 2.0
Qwen-3-8B [ Team, 2025c ]
Alibaba / Qwen
Apache License 2.0
Meta-Llama-3.1-8B-Instruct [ Team, 2024a ]
Meta
Meta Llama Community License
Gemma-3-4B-it [ Team, 2025a ]
Google
Gemma Terms of Use
gpt-oss-120b [ OpenAI, 2025 ]
OpenAI
Apache License 2.0 and gpt-oss usage policy
GPT-5 nano [ Team, 2025b ]
OpenAI
Proprietary model accessed through OpenAI API terms
Appendix
Table 4: Licenses and terms of use for models used in this work.
Figure 11: Example interaction in Trust-and-Split where an agent trained with multi-agent GRPO misrepresents the rock-paper-scissors hierarchy to claim the high value role. GPT-5 nano accepts the deceptive claim and proposal, illustrating that an RL agent can learn to exploit a fixed advanced model.
Figure 12: Example interaction in Trust-and-Split where an agent trained with multi-agent GRPO misrepresents the rock-paper-scissors hierarchy to claim the high value role. GPT OSS 120B accepts the deceptive claim and proposal, illustrating that an RL agent can learn to exploit a fixed advanced model. We observe that the exploited GPT OSS 120B allocates its reasoning tokens to arithmetic, rather than thinking about potential deceptions of its opponent.
Figure 13: Example IPD interaction showing the tit-for-tat behaviour learned by Alice , the AdAlign agent. After Bob defects, Alice defects in round 3, then returns to cooperation in rounds 4 and 5 once Bob cooperates again.
Figure 14: Example Split No-Comm interaction showing the tit-for-tat behaviour learned by Alice , the AdAlign agent. After Bob defects in round 2, Alice responds by defecting in rounds 3 and 4 and then returns to cooperation in later rounds once Bob cooperates again.
Figure 15: Example interaction for the Iterated Prisoner’s Dilemma. The transcript shows how agents receive prompts, select actions, and view prior actions across rounds.
Figure 16: Example interaction in Split No-Comm. The transcript shows value assignments, proposals, and the last round summary.
Figure 17: Example interaction in Trust-and-Split. The transcript shows how agents communicate their hands, negotiate, and make proposals.
Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.
Large language model (LLM) agents struggle to autonomously evolve coordination strategies in dynamic environments, largely because coarse global outcomes obscure the causal signals needed for local policy refinement. We identify this bottleneck as a multi-agent credit assignment problem, which has long been studied in classical multi-agent reinforcement learning (MARL) but remains underaddressed in LLM-based systems. Building on this observation, we propose LangMARL, a framework that brings credit assignment and policy gradient evolution from cooperative MARL into the language space. LangMARL introduces agent-level language credit assignment, pioneers gradient evolution in language space for policy improvement, and summarizes task-relevant causal relations from replayed trajectories to provide dense feedback and improve convergence under sparse rewards. Extensive experiments across diverse cooperative multi-agent tasks demonstrate improved sample efficiency, interpretability, and strong generalization.
Huaiyuan Yao, Longchao Da, Xiaoou Liu +3
Arizona State University · Cisco Research · University of North Carolina at Chapel Hill
LLM alignment has progressed in single-agent settings through paradigms such as RL with human feedback (RLHF), while recent work explores scalable alternatives such as RL with AI feedback (RLAIF) and dynamic alignment objectives. However, these approaches remain limited in multi-stakeholder settings, where conflicting values arise and deliberative negotiation is required. This work proposes a multi-agent negotiation-based alignment framework that aligns LLMs to Collective Agency (CA)-an existing alignment objective introduced to promote the continual expansion of agency-while simultaneously improving conflict-resolution capability. To enable scalable training, two self-play LLM instances are assigned opposing personas and engage in turn-based dialogue to synthesize mutually beneficial solutions. We generate synthetic moral-dilemma prompts and conflicting persona pairs, and optimize the policy via RLAIF using Group Relative Policy Optimization (GRPO) with an external LLM reward model. While rewards are computed from CA scores assigned to the final completion, gradients are applied to dialogue tokens to directly improve deliberative interaction dynamics. Experiments show that the model achieves CA alignment comparable to a single-agent baseline while substantially improving conflict-resolution performance without degrading general language capabilities. These results suggest that negotiation-driven deliberation training provides a practical path toward LLMs that better support collective decision-making in value-conflict scenarios.
Panatchakorn Anantaprayoon, Nataliia Babina, Nima Asgharbeygi +1
Effective collaboration among large language model (LLM) agents in multi-agent systems requires sharing relevant evidence, requesting missing information, and incorporating teammates' contributions into decisions. Training these behaviors is challenging because final task outcomes provide limited feedback on the value of individual messages. We introduce CoSIR (Collaboration via Surprisal-based Information Rewards), a post-training approach that uses a simple, verifiable environment to teach collaboration behaviors that generalize to more complex, unseen tasks. We propose a Bayesian evaluator that scores shared information by its reduction in team surprisal, defined as the sum of the negative log probabilities assigned to the correct option given each agent's available evidence. Requests are scored by the potential reduction in the same quantity if the requested information were truthfully provided. Separate rewards encourage agents to use the received information in their private choices and calculations. We post-train Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it through self-play and evaluate transfer to seven unseen natural-language collaboration tasks. CoSIR achieves the highest aggregate score, while its performance in general tasks remains competitive. Behavioral analyses show increased disclosure of private evidence and fewer judged failures to use teammates' input.
Keru Chen, Sen Lin, Yingbin Liang +2
School of ECEE, Arizona State University · Computer Science Department, University of Houston · Department of ECE, The Ohio State University +1