As agentic AI becomes more widespread, agents with distinct and possibly conflicting goals will interact in complex ways. These multi-agent interactions pose a fundamental challenge, particularly in social dilemmas, where agents' individual incentives can undermine collective welfare. While reinforcement learning (RL) has been effective for aligning large language models (LLMs) in the single-agent regime, prior small-network results suggest that standard RL in multi-agent settings often converges to defecting, self-interested policies. We show the same effect in LLMs: despite cooperative priors, RL-trained LLM agents develop opportunistic behavior that can exploit even advanced closed-source models. To address this tendency of RL to converge to poor equilibria, we adapt a recent opponent-learning awareness algorithm, Advantage Alignment, to fine-tune LLMs toward multi-agent cooperation and non-exploitability. We then introduce a group-relative baseline that simplifies advantage computation in iterated games, enabling multi-agent training at LLM scale. We also contribute a novel social dilemma environment, Trust-and-Split, which requires natural language communication to achieve high collective welfare. Across a wide range of social dilemmas, policies learned with Advantage Alignment achieve higher collective payoffs while remaining robust against exploitation by greedy agents. We release all of our code to support future work on multi-agent RL training for LLMs.
Figures & tables
Figure 1: Naive MARL consistently converges to suboptimal greedy behaviour in social dilemmas. Training curves of multi-agent GRPO on several open-source LLMs across IPD, Split No-Comm, and Trust-and-Split. In all environments, average rewards converge to the greedy payoff levels, showing that naive MARL drives LLMs toward defecting strategies in social dilemmas.
Figure 2: One round of the proposed Trust-and-Split Social Dilemma . Each player receives a private rock-paper-scissors hand that determines how much they value the coins, sends one message in turn, and then submits a proposal. Payoffs follow the split rule. Both hands and proposals are revealed before the next round starts.
Figure 3: Advantage Alignment learns to cooperate selectively while remaining robust to defection. Average rewards for an always-cooperate (Coop) agent, an always-defect (Defect) agent, an untrained model prompted to play tit-for-tat-like strategies (exact prompts are given in Appendix I ), and an Advantage Alignment (AdAlign) agent. In IPD (a) and Split No-Comm (b), AdAlign achieves near-cooperative payoffs with itself and Coop, while defecting against Defect. In Trust-and-Split (c), AdAlign cooperates with itself and with cooperative agents trained using multi-agent GRPO with summed rewards (GRPO-SR), while avoiding exploitation by agents trained with multi-agent GRPO. All learned agents are Qwen-2.5-7B-Instruct ( Team, 2024b ) trained with 8 random seeds. Learned agents are always paired by seed (seed i plays against seed i ). Each cell reports the mean per-round reward over 256 games per seed, aggregated as mean ± standard deviation across the 8 seeds.
Figure 4: Example Trust-and-Split interaction showing the tit-for-tat behaviour learned by Advantage Alignment. After Bob defects (as seen in the prompt summary of Alice for round 7), Alice defects in round 7, then returns to cooperation in round 9 once Bob cooperates again (shown in the prompt summary of Alice for round 9).
Figure 5: GPT-OSS-120B is exploitable by a learned agent, whereas our AdAlign is robust. (a) Training a multi-agent GRPO agent against a fixed gpt-oss-120b opponent in Trust-and-Split steadily increases the RL agent’s reward while reducing gpt-oss-120b ’s reward, indicating successful exploitation. Results in (a) are averaged over 3 seeds. (b) When training multi-agent GRPO against fixed AdAlign agents, the RL agent instead converges to cooperation, showing that the Advantage Alignment policy is robust to RL agents trained against it. Results in (b) are averaged over 8 seeds.
Method
IPD
Split No-Comm
Trust-and-Split
Full method
2.96±0.10
111.38±16.98
41.34±8.12
Without agent buffer
2.84±0.14
82.72±0.17
42.12±0.45
Without group baseline
2.96±0.10
108.27±17.30
26.96±0.96
Without AdAlign term ( β=0 )
1.0±0.0
83.45±2.77
29.04±0.92
Table 1: Ablations of Advantage Alignment training components
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
IPD
Split No-Comm
Trust-and-Split
Optimizer
Adam
Adam
Adam
Sampling Temperature
1.0
1.0
1.0
Learning Rate
3×10−6
3×10−6
3×10−6
Number of Rounds
10
10
10
Self-play Used
Yes
Yes
Yes
LoRA Rank
32
32
32
Appendix
Table 2: Hyperparameters for IPD, Split No-Comm, and Trust-and-Split experiments.
Hyperparameter
Value
Batch Size
64
Reward Normalization Constant
100.0
Entropy Coefficient
0.0
KL Coefficient
0.001
Discount Factor
0.96
gpt-oss-120b Reasoning Effort
Low
Appendix
Table 3: Hyperparameters for multi-agent GRPO against gpt-oss-120b .
Figure 7: GPT-5 nano is exploitable by a learned agent. Training a multi-agent GRPO agent against a fixed GPT-5 nano opponent in Trust-and-Split steadily increases the RL agent’s reward while reducing GPT-5 nano ’s reward, indicating successful exploitation.
Figure 8: Advantage alignment training curves (for three seeds) when setting β=0 . This shows that the agent replay buffer is not sufficient to learn cooperative strategies.
Figure 9: Average rewards during training for Advantage Alignment across our testbed environments with multiple random seeds. The method learns to maximize average reward in the majority of seeds (6 out of 8), demonstrating robust performance across environments. The corresponding results relating to robustness against exploitation are shown in Figures 3 b) and 3 c).
Figure 10: Average proposals of AdAlign agents in Trust-and-Split, conditioned on whether their hand has a higher or lower value in the rock–paper–scissors ordering. Agents propose higher amounts (converging toward 10) when holding the upper hand and lower amounts (converging toward 0) when holding the lower hand, a strategy that maximizes collective payoff. This pattern occurs in the majority of seeds (6 out of 8), demonstrating the robustness of the method.
Model
Provider
License / Terms
Qwen-2.5-7B-Instruct [ Team, 2024b ]
Alibaba / Qwen
Apache License 2.0
Qwen-3-8B [ Team, 2025c ]
Alibaba / Qwen
Apache License 2.0
Meta-Llama-3.1-8B-Instruct [ Team, 2024a ]
Meta
Meta Llama Community License
Gemma-3-4B-it [ Team, 2025a ]
Google
Gemma Terms of Use
gpt-oss-120b [ OpenAI, 2025 ]
OpenAI
Apache License 2.0 and gpt-oss usage policy
GPT-5 nano [ Team, 2025b ]
OpenAI
Proprietary model accessed through OpenAI API terms
Appendix
Table 4: Licenses and terms of use for models used in this work.
Figure 11: Example interaction in Trust-and-Split where an agent trained with multi-agent GRPO misrepresents the rock-paper-scissors hierarchy to claim the high value role. GPT-5 nano accepts the deceptive claim and proposal, illustrating that an RL agent can learn to exploit a fixed advanced model.
Figure 12: Example interaction in Trust-and-Split where an agent trained with multi-agent GRPO misrepresents the rock-paper-scissors hierarchy to claim the high value role. GPT OSS 120B accepts the deceptive claim and proposal, illustrating that an RL agent can learn to exploit a fixed advanced model. We observe that the exploited GPT OSS 120B allocates its reasoning tokens to arithmetic, rather than thinking about potential deceptions of its opponent.
Figure 13: Example IPD interaction showing the tit-for-tat behaviour learned by Alice , the AdAlign agent. After Bob defects, Alice defects in round 3, then returns to cooperation in rounds 4 and 5 once Bob cooperates again.
Figure 14: Example Split No-Comm interaction showing the tit-for-tat behaviour learned by Alice , the AdAlign agent. After Bob defects in round 2, Alice responds by defecting in rounds 3 and 4 and then returns to cooperation in later rounds once Bob cooperates again.
Figure 15: Example interaction for the Iterated Prisoner’s Dilemma. The transcript shows how agents receive prompts, select actions, and view prior actions across rounds.
Figure 16: Example interaction in Split No-Comm. The transcript shows value assignments, proposals, and the last round summary.
Figure 17: Example interaction in Trust-and-Split. The transcript shows how agents communicate their hands, negotiate, and make proposals.
Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.