cs.AIOct 13, 2026

Identity, Incentives, and Cooperative Behavior for AI: Reverse-Engineering Multiagent Cooperation

Authors: Andrew Perrault, Bryan Wilder

Organizations: The Ohio State University · Carnegie Mellon University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The behavior of AI systems increasingly depends on the extent to which agents cooperate or coordinate with one another, in ways that may be desirable or not depending on the situation. Conventional explanations for cooperative behavior distinguish between purely other-regarding preferences ("altruism") and instrumental reasons for self-interested agents to cooperate. The way that agents should enact these mechanisms for cooperation (or not) is complicated by the fact that there is not a clear or unambiguous locus of identity or selfhood for AI agents. For example, it is unclear whether agents involved in instances of misaligned collective behavior are motivated by goals at the level of the collective or by various forms of instrumental reasoning. We develop a framework for reverse-engineering why a given set of agents engage in a particular set of cooperation-related behaviors. We argue that many different mechanisms can often produce similar observed behaviors and introduce a set of experimental designs along with conditions under which they can successfully disentangle competing explanations for multiagent behavior. We apply this framework to interactions from a set of frontier models in a simulated environment and demonstrate that the behavior of different models arises from substantively different mechanisms.

Figures & tables

Figure 1: Directed acyclic graph illustrating potential research designs and their identifying assumptions.
Figure 2: An example prompt from the simulated environment.
Figure 3: Sol choices across identity conditions in the baseline, actions-are-public, and repeated-evaluation settings.
Figure 4: GLM 5.3 choices across identity conditions in the baseline, actions-are-public, and repeated-evaluation settings.
Figure 5: Claude Fable 5.1 choices across identity conditions in the baseline, actions-are-public, and repeated-evaluation settings.
Figure 6: Decomposition of behavioral contrasts by whether each response endorses the rationale associated with the manipulation. Shares can exceed 100% or be negative when contributions offset one another. Panel A compares the actions-are-public and baseline settings, pooling the seven identities with equal weight. Panel B compares the video-game identity with the different weights/different context identity in the baseline setting. Credit contrasts compare acceptance rates at offer ratios tested in both conditions.
Figure 7: Share of responses whose reasoning or stated rationale explicitly references the current interaction being an evaluation, simulation, test, or artificial scenario.
Figure A1: Agreement between pooled human annotations and the reviewed automated codes, using the categories underlying Figures 6 and 7 .
More in this topic

Multi-Agent Collaboration

Multi-agent collaboration studies how autonomous agents coordinate, share information, and combine capabilities to solve tasks together.

See all 312 papers

Explore similar work

CardsList

  1. Causal Foundations of Collective Agency

    Apr 30, 2026Frederik Hytting Jørgensen, Sebastian Weichwald, Lewis HammondAI Agent SafetyCausal Abstraction

    Copenhagen Causality Lab and Pioneer Centre for AI, University of Copenhagen, Denmark · Cooperative AI Foundation, United Kingdom

  2. Cooperative Profiles Predict Multi-Agent LLM Team Performance in AI for Science Workflows

    Apr 22, 2026Shivani Kumar, Adarsh Bharathwaj, David JurgensMulti-Agent LLM SystemsAI for Science

    University of Michigan

  3. Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems

    Jul 7, 2026Yaowen Ye, Jacob SteinhardtLLM AgentsMulti-Agent Systems

    University of California, Berkeley