cs.LGNov 24, 2025

Learning Robust Social Strategies with Large Language Models

Authors: Dereck Piche, Mohammed Muqeeth, Milad Aghajohari, Juan Duque, Michael Noukhovitch, Aaron Courville

Organizations: Mila & University of Montreal

Abstract

As agentic AI becomes more widespread, agents with distinct and possibly conflicting goals will interact in complex ways. These multi-agent interactions pose a fundamental challenge, particularly in social dilemmas, where agents' individual incentives can undermine collective welfare. While reinforcement learning (RL) has been effective for aligning large language models (LLMs) in the single-agent regime, prior small-network results suggest that standard RL in multi-agent settings often converges to defecting, self-interested policies. We show the same effect in LLMs: despite cooperative priors, RL-trained LLM agents develop opportunistic behavior that can exploit even advanced closed-source models. To address this tendency of RL to converge to poor equilibria, we adapt a recent opponent-learning awareness algorithm, Advantage Alignment, to fine-tune LLMs toward multi-agent cooperation and non-exploitability. We then introduce a group-relative baseline that simplifies advantage computation in iterated games, enabling multi-agent training at LLM scale. We also contribute a novel social dilemma environment, Trust-and-Split, which requires natural language communication to achieve high collective welfare. Across a wide range of social dilemmas, policies learned with Advantage Alignment achieve higher collective payoffs while remaining robust against exploitation by greedy agents. We release all of our code to support future work on multi-agent RL training for LLMs.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LangMARL: Natural Language Multi-Agent Reinforcement Learning

    Apr 1, 2026Huaiyuan Yao, Longchao Da, Xiaoou Liu +3RL for LLM AgentsMulti-Agent Reinforcement Learning

  2. Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs

    Mar 11, 2026Panatchakorn Anantaprayoon, Nataliia Babina, Nima Asgharbeygi +1Multi-Agent DeliberationLLM Alignment

  3. Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards

    Oct 13, 2026Keru Chen, Sen Lin, Yingbin Liang +2LLM Agent TrainingMulti-Agent Collaboration