In Trust We Survive: Emergent Trust Learning

Qianpu Chen; Giulio Barbero; Mike Preuss; Derya Soydaner

cs.MAMar 18, 2026

In Trust We Survive: Emergent Trust Learning

Authors: Qianpu Chen, Giulio Barbero, Mike Preuss, Derya Soydaner

Organizations: LIACS, Leiden Institute of Advanced Computer Science, Universiteit Leiden, The Netherlands

Abstract

We introduce Emergent Trust Learning (ETL), a lightweight, trust-based control algorithm that can be plugged into existing AI agents. It enables these to reach cooperation in competitive game environments under shared resources. Each agent maintains a compact internal trust state, which modulates memory, exploration, and action selection. ETL requires only individual rewards and local observations and incurs negligible computational and communication overhead. We evaluate ETL in three environments: In a grid-based resource world, trust-based agents reduce conflicts and prevent long-term resource depletion while achieving competitive individual returns. In a hierarchical Tower environment with strong social dilemmas and randomised floor assignments, ETL sustains high survival rates and recovers cooperation even after extended phases of enforced greed. In the Iterated Prisoner's Dilemma, the algorithm generalises to a strategic meta-game, maintaining cooperation with reciprocal opponents while avoiding long-term exploitation by defectors. Code will be released upon publication.

Figures & tables

Panel of Figure 1 — Fig. 2 : Schematic of the memory mechanism in ETL, highlighting how short-term and long-term experience buffers are updated from state-action-reward tuples and jointly drive subsequent action decisions.

Panel of Figure 2 — Fig. 3 : Overview of the trust mechanism in ETL: locally observed cooperation and dissatisfaction signals update a scalar trust state, which is smoothed over time and fed back into decision making without requiring explicit communication or identities.

Panel of Figure 3 — Fig. 4 : Exploration mechanism in ETL, where the exploration rate is adapted based on trust stability, perceived risk and recent performance, so that agents explore more under uncertainty and exploit more in stable, trusted regimes.

Fig. 5 : ETL reduces long-run conflict intensity compared to a trust-free baseline, yielding substantially fewer low-value clashes over shared resources across 30 independent runs.

Fig. 6 : ETL stabilises the environment by keeping the fraction of tiles in cooldown consistently lower than a trust-free baseline, indicating less persistent over-exploitation of shared resources.

Fig. 7 : Total remaining resources at the end of each episode in the grid environment (mean over 30 seeds); runs with ETL leave substantially more resources on the map than the baseline while optimising the same individual reward.

Fig. 8 : Initial state of the Tower environment. The food platform starts at the top of the tower, and agents are randomly assigned to floors.

Fig. 9 : Success rate in the Tower environment over training episodes, comparing ETL to tabular Q-Learning and Monte Carlo control. Only ETL achieves and maintains a high success rate without explicit cooperative rewards.

Fig. 10 : Recovery of ETL in the Tower environment after 200 episodes of forced greedy behaviour: the success rate starts near zero during enforced greed but climbs back above 90% once learning resumes, demonstrating robustness to unfavourable initial conditions.

Player A	Player B	Payoff (A, B)
Cooperate (C)	Cooperate (C)	(3, 3)
Cooperate (C)	Defect (D)	(0, 5)
Defect (D)	Cooperate (C)	(5, 0)
Defect (D)	Defect (D)	(1, 1)

TABLE I : Payoff matrix of the Iterated Prisoner’s Dilemma used in our experiments, with the standard ordering where unilateral defection yields the highest individual payoff but mutual cooperation dominates mutual defection.

Fig. 11 : Success rate across all opponents in the Iterated Prisoner’s Dilemma round-robin tournament; ETL outperforms classic strategies such as Tit-for-Tat, Always Cooperate and Always Defect, including against delayed-response adversaries that attempt to exploit its trust mechanism.

Multi-Agent Coordination

Multi-agent coordination studies how autonomous agents align actions, share information, and divide work to achieve individual or collective goals.

See all 516 papers

Trust Between AI Agents: Measuring Formation, Breakage, and Recovery, with Implications for Governing Multi-Agent Systems
Jun 12, 2026Yujiao ChenAI Agent Evaluation Multi-Agent Systems
Massachusetts Institute of Technology Cambridge, MA 02139
Trust-Gated Capability Control: Breaking the Trust-Vulnerability Paradox in Multi-Agent LLM Systems
Oct 4, 2026Mehedi Hasan Nipu, Chinmoy Mitra, Tarannum Ahmed Nowshin +1Trust in AI Multi-Agent System Security
Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh · Department of Computer Science & Engineering Rajshahi University of Engineering & Technology Rajshahi, Bangladesh · Department of Computer Science and Engineering BRAC University,Dhaka, Bangladesh +1
The Energy Society: A Simulation Environment for Studying Agent Cooperation under Survival Pressure
Jul 16, 2026Lucas Bergholdt Hansen, Federico Torrielli, Filippo Tonini +1Social Dilemmas Multi-Agent Coordination
University of Southern Denmark Odense, Denmark · University of Turin Torino, Italy

Jun 12, 2026J/K move · Enter open · S save

Trust Between AI Agents: Measuring Formation, Breakage, and Recovery, with Implications for Governing Multi-Agent Systems

Yujiao Chen

Massachusetts Institute of Technology Cambridge, MA 02139

AI Agent Evaluation Multi-Agent Systems Trust in AI

Open paper AskPDF arXiv

As language-model agents increasingly work in teams, each agent must decide how much to trust its teammates. Yet we lack a standard way to measure trust between AI agents. We propose a behavioral measure based on costly verification. In a cooperative survival game, checking a teammate's work consumes resources, while trusting a wrong answer can be fatal. Relative to a memoryless version of the same model, reduced verification provides an observable measure of trust. Using this framework, we study trust formation, breakage, and recovery across six frontier model snapshots. When paired with a consistently reliable teammate, four snapshots (Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.1, and Gemini 3.1 Pro) reduce verification by roughly 60-85%, whereas two smaller snapshots show little or no such adjustment. Failures reverse this discount, but models differ in how they respond. Some concentrate renewed scrutiny on the culprit, while others become more cautious toward the entire team. Recovery is slower than formation, and clustered failures sustain suspicion far longer than the same number of failures spread apart. These differences have practical consequences. Models that form trust verify less, decide more quickly, and achieve higher payoffs in our environment. By contrast, persistent over-verification is associated with indecision rather than safety. Our results show that trust dispositions can be measured before deployment and suggest that calibration, rather than maximal suspicion, should be the central concern in the governance of multi-agent AI systems.

Abstract

Multi-Agent Coordination

Explore similar work

Trust Between AI Agents: Measuring Formation, Breakage, and Recovery, with Implications for Governing Multi-Agent Systems

Trust-Gated Capability Control: Breaking the Trust-Vulnerability Paradox in Multi-Agent LLM Systems

The Energy Society: A Simulation Environment for Studying Agent Cooperation under Survival Pressure