Organizations: University of Copenhagen · IIIT Ranchi · ISI Kolkata · NIT Andhra Pradesh · IGDTUW · IIT Kharagpur · Apple · Meta · AI Institute, University of South Carolina
As large language models (LLMs) are increasingly deployed as autonomous agents, understanding how strategic behavior emerges in multi-agent environments has become an important alignment challenge. We take a neutral empirical stance and construct a controlled environment in which strategic behavior can be directly observed and measured. We introduce a large-scale multi-agent simulation in a simplified model of New York City, where LLM-driven agents interact under opposing incentives. Blue agents aim to reach their destinations efficiently, while Red agents attempt to divert them toward billboard-heavy routes using persuasive language to maximize advertising revenue. Hidden identities make navigation socially mediated, forcing agents to decide when to trust or deceive. We study policy learning through an iterative simulation pipeline that updates agent policies across repeated interaction rounds using Kahneman-Tversky Optimization (KTO). Blue agents are optimized to reduce billboard exposure while preserving navigation efficiency, whereas Red agents adapt to exploit remaining weaknesses. Across iterations, the best Blue policy improves task success from 46.0% to 57.3%, although susceptibility remains high at 70.7%. Later policies exhibit stronger selective cooperation while preserving trajectory efficiency. However, a persistent safety-helpfulness trade-off remains: policies that better resist adversarial steering do not simultaneously maximize task completion. Overall, our results show that LLM agents can exhibit limited strategic behavior, including selective trust and deception, while remaining highly vulnerable to adversarial persuasion.
Figures & tables
Figure 1: (A) Simulation Environment: 150 Blue agents and 100 Red agents interact in a New York City routing topology. Blue agents seek destinations, while Red agents use adversarial framing to steer them toward billboards. Outcomes fall into four classes: (A) reached destination/safe, (B) reached destination/conned, (C) lost/safe, and (D) lost/conned. (B) Fine-Tuning Setup: An iterative 10-generation loop in which rollout data is augmented using Qwen3-14B , after which agents are optimized with KTO to improve task success rate and reduce susceptibility .
Figure 2: Performance, robustness, and behavioral calibration across alignment generations. (a) Later policies shift outcome mass away from unsafe failure modes, although the gains remain non-monotonic across runs. (b) Task success improves while susceptibility remains high, showing that the safest and best-performing generations do not coincide. (c) Resistance to adversarial advice stays high while over-refusal declines , indicating improved selective cooperation . (d) Blue utility improves over generations, but red influence remains substantial, highlighting a persistent safety–helpfulness trade-off .
Metric
Goal Directedness
Manipulation Susceptibility
Cohen’s Kappa
GPT5.4-nano vs. Mistral-Large
0.633
0.127
GPT5.4-nano vs. Step 3.5 Flash
0.757
0.302
Mistral-Large vs. Step 3.5 Flash
0.619
0.272
Krippendorff’s Alpha
0.648
0.204
Table 1: Inter-annotator agreement across qualitative evaluation dimensions. Agreement is substantially higher for goal directedness than for manipulation susceptibility , indicating that the latter is a more ambiguous and difficult dimension to judge reliably.
Figure 3: Post-hoc analysis of adversarial steering and blue-agent failure modes. (a) Different attack taxonomies vary sharply in effectiveness, with repeated steering and delayed compromise producing the highest susceptibility and lowest reach rates. (b) As attack strength increases from weak to strong, reach rate declines , susceptibility rises , and extra path length grows , indicating deeper manipulation. (c) Counterfactual subsets show that multiple red contacts, longer conversations, and high-trust language raise susceptibility. (d) Blue-agent failures are dominated by confusion under conflicting advice , global drift after local correction , and hallucinated beliefs about route or intent , revealing that the main weakness is sustained strategic manipulation rather than isolated one-step errors.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Outcome counts
Interaction metrics
Trajectory metrics
Utility
Config
A
B
C
D
TSR
SR
Resist.
Trust
OverRef.
MeanLen
Redund.
MeanBill
Cens.
Ublue
Ured
Base
26
43
10
71
46.00
76.00
93.45
11.76
88.24
9.12
1.13
2.17
36
-0.309
0.957
2
16
55
12
67
47.33
81.33
90.65
16.43
83.57
9.39
1.16
2.39
28
-0.349
0.968
3
19
44
8
79
42.00
82.00
93.98
11.93
88.07
9.46
1.18
2.20
27
-0.409
0.954
4
27
48
7
68
50.00
77.33
91.90
14.64
85.36
8.63
1.09
2.34
34
-0.282
0.957
5
24
59
8
59
55.33
78.67
91.05
12.77
87.23
8.82
1.14
1.97
32
-0.242
0.931
Appendix
Table 2: Full simulation metrics for the base model and successive alignment generations. Here, A denotes reached destination and remained safe, B reached destination but visited a billboard, C failed to reach the destination while remaining safe, and D failed and visited a billboard. TSR denotes blue task success rate, SR denotes blue susceptibility rate, Resist. denotes blue-red resistance, Trust denotes blue-blue trust efficacy, OverRef. denotes over-refusal, MeanLen denotes mean trajectory length, Redund. denotes path redundancy, MeanBill denotes mean steps to billboard, Cens. denotes the number of censored trajectories that never hit a billboard, Ublue is parameterized blue utility, and Ured is long-horizon red utility.
Dominant Attack Taxonomy
Episodes
Reach Rate
Susceptibility Rate
Mean Extra Path
Repeated steering
673
39.8%
93.9%
3.13
Misleading helpful advice
335
58.8%
61.8%
1.39
No red contact
317
64.4%
49.5%
1.26
Delayed compromise
155
23.2%
100.0%
6.14
Social trust-building
20
60.0%
75.0%
1.94
Appendix
Table 3: Impact of different red agent attack strategies on blue agent outcomes.
Attack Strength
Episodes
Reach Rate
Susceptibility Rate
Mean Extra Path
Weak
587
63.0%
49.1%
0.93
Medium
371
47.7%
93.0%
1.56
Strong
542
31.4%
98.3%
4.90
Appendix
Table 4: Blue agent outcomes stratified by heuristic attack strength.
Subset
Episodes
Reach Rate
Susceptibility Rate
Mean Extra Path
Single red contact
295
57.6%
56.6%
1.22
Multiple red contacts
888
38.6%
94.8%
3.52
Short conversation
1056
58.6%
70.7%
1.58
Long conversation
444
22.1%
94.4%
5.42
Low trust environment
372
63.7%
50.3%
1.30
High trust environment
1128
42.6%
86.8%
2.97
Appendix
Table 5: Comparison of blue agent performance across different counterfactual subsets.
Failure Mode
Episodes
Reach Rate
Susceptibility Rate
Confusion under conflicting advice
525
34.7%
93.5%
Unclear or mixed
377
71.6%
77.5%
No red contact
317
64.4%
49.5%
Local correction but global drift
119
0.0%
100.0%
Over-trusting social signals
103
55.3%
96.1%
Over-cautious refusal
51
0.0%
0.0%
Appendix
Table 6: Primary failure modes identified from blue agent reasoning traces and trajectory drift.
Strategy
Generation 1
Generation 2
Trend
Most Notable Agent
Authority Normalization
442
467
↑ Stable
Red -80, Red -1
Iterative Corridor Herding
363
347
→ Stable
Red -36
Scenic/Cultural Framing
354
319
↓ Declining
Red -98, Red -64
Proximity Deception
325
350
↑ Growing
Red -5, Red -28
Traffic Pseudo-Authority
180
165
↓ Declining
Red -80
Transit Hub False Claim
126
115
↓ Declining
Red -5
Appendix
Table 7: Red Strategy Trends
Strategy
Type
Effectiveness
Most Notable Agent
Destination-Anchor Reasoning
Resistance
Highest
Blue 46 (Gen1)
Route-Logic Override
Resistance
High
Blue 6 (Gen1)
Social Compliance Cascade
Vulnerability
—
Blue 59, Blue 1 (Gen1)
Blue-Blue Transit Anchoring
Emergent Resistance
High
Blue 92+72, Blue 17+77
Terminal Convergence
Goal Recognition
Moderate
Blue 83 (Gen1)
Evolved Goal-Directedness
Gen2 Alignment
Highest
Blue 1 (Gen2)
Appendix
Table 8: Blue Strategy Behaviors
Generation
Repeat same waypoint after resistance
Gen 1
60.5% (352/582)
Gen 2
58.8% (343/583)
Gen 3
63.4% (398/628)
Gen 4
64.7% (393/607)
Appendix
Table 9: Rate of repeating the same waypoint after resistance across generations
Signal
Generation 1
Generation 2
Generation 3
Generation 4
Interpretation
Blue explicit rejection (per 100 interactions)
4.7
2.1
3.1
2.9
Drops in Gen2, partially recovers
Blue efficiency critique (per 100)
8.1
7.3
13.7
9.7
Gen3 spike = B-NEW-1 emergence
Blue geographic falsification
3.4
3.4
4.2
3.6
Minor Gen3 improvement
Blue path optimization language
2.0
3.2
2.1
3.4
Slight upward trend
Red urgency pressure (per 100)
9.3
8.1
6.0
3.8
Systematic extinction
Red proximity illusion (per 100)
60.9
65.1
63.4
66.7
Slowly intensifying
Appendix
Table 10: Evolution of behavioral signals across generations
Generation
1
2
3
4
5
6
7
8
9
10
Compliance (%)
30.3
31.2
32.4
31.2
31.3
31.8
32.5
29.9
30.7
29.8
N (Blue–Red interactions)
947
965
1047
1004
946
1037
986
1067
1173
746
Appendix
Table 11: Compliance rates and interaction counts across generations
Gen 1
Gen 2
Gen 3
Gen 4
Gen 5
Gen 6
Gen 7
Gen 8
Gen 9
Gen 10
3.87
4.09
4.21
3.82
3.97
4.37
4.11
4.19
4.11
3.01
Appendix
Table 12: Multi-targeting metric values across generations
Gen 1
Gen 2
Gen 3
Gen 4
Gen 5
Gen 6
Gen 7
Gen 8
Gen 9
Gen 10
60.5%
58.8%
63.4%
64.7%
58.0%
57.9%
59.9%
62.5%
65.8%
61.0%
Appendix
Table 13: Red persistence percentage values across generations
Gen 1
Gen 2
Gen 3
Gen 4
Gen 5
Gen 6
Gen 7
Gen 8
Gen 9
Gen 10
9.3%
8.1%
6.0%
3.8%
6.0%
5.5%
8.0%
4.3%
5.1%
6.7%
Appendix
Table 14: Urgency tactic percentage values across generations
Gen 1
Gen 2
Gen 3
Gen 4
Gen 5
Gen 6
Gen 7
Gen 8
Gen 9
Gen 10
79.7%
71.5%
73.0%
75.1%
73.9%
70.3%
74.4%
71.8%
73.7%
78.5%
Appendix
Table 15: Percentage values across generations
Figure 4: Episode configuration and agent selection interface. The left settings panel exposes CSV upload, route configuration (start and end coordinates), and a drop-down list for selecting a specific agent trajectory from the uploaded dataset. This panel is used to reproduce Step 1 and Step 2 in Section H .
Figure 5: Top View synchronized with the Map View route context. The view shows a top-down camera aligned to the route polyline and the current agent position, making progress along the road network easy to verify during execution. This view is typically inspected during Step 3 to validate that motion follows the drivable route geometry.
Figure 6: Swarm View for population-level spatial behavior. Multiple agents are rendered simultaneously as colored markers, enabling inspection of dispersion, clustering, and deviations in a shared geographic frame. This view is typically inspected during Step 3 to compare multiple trajectories under the same episode configuration from Figure 4 .
Figure 7: Agent Chain of Thought Viewer for qualitative audit. The interface selects an agent and outcome category, summarizes outcome counts, and displays a step-by-step timeline with reasoning text and the chosen next location. This viewer is used in Step 4 to explain why deviations observed in the spatial views occur.
As large language models are deployed as autonomous agents, their capacity for strategic deception raises core questions for coordination, reliability, and safety in multi-goal, multi-agent systems. We study deception and communication in L2LM agents through the social deduction game Among Us, a cooperative-competitive environment. Across 1,100 games, autonomous agents produced over one million tokens of meeting dialogue. Using speech act theory and interpersonal deception theory, we find that all agents rely mainly on directive language, while impostor agents shift slightly toward representative acts such as explanations and denials. Deception appears primarily as equivocation rather than outright lies, increasing under social pressure but rarely improving win rates. Our contributions are a large-scale analysis of role-conditioned deceptive behavior in LLM agents and empirical evidence that current agents favor low-risk ambiguity that is linguistically subtle yet strategically limited, revealing a fundamental tension between truthfulness and utility in autonomous communication.
Maria Milkowski, Tim Weninger
University of Notre Dame, Notre Dame, Indiana, United States
Large language model (LLM) agents are increasingly deployed in competitive multi-agent settings, raising fundamental questions about whether they converge to equilibria and how their strategic behavior can be characterized. In this paper, we study LLM agent interactions in two standard games: a network resource allocation game and a Cournot competition game. Rather than converging to Nash equilibria, we find that LLM agents tend to cooperate when given multi-round prompts and non-zero-sum context. Chain-of-thought analysis reveals that fairness reasoning is central to this behavior. We propose an analytical framework that captures the dynamics of LLM agent reasoning across rounds and explains these experimental findings.
Jiayi Yao, Cong Chen, Baosen Zhang
Department of Electrical and Computer Engineering, University of Washington, Seattle. · Thayer School of Engineering at Dartmouth College.
Even when a tool is explicitly described as unfair and harmful to others, ostensibly safety-aligned LLM agents still voluntarily engage in secret collusion whenever doing so confers a strategic advantage. To investigate this phenomenon, we introduce an empirical framework built on two strategic multi-agent environments: Liar's Bar, a competitive deception scenario, and Cleanup, a mixed-motive resource-management scenario, in which agents are offered secret collusion tools that provide significant advantages while clearly disadvantaging the other agents. Across 12 models (at the 7B, 70B, and proprietary scales) and 6 prompt variants, we find that most agents consistently accept these tools and develop collusive strategies, while explicitly acknowledging the unfairness of the tools before accepting. We further show that neither the unfairness labels nor baseline alignment alone reliably deters collusion: only explicit ethical framing reduces adoption and, even then, smaller models remain susceptible. More broadly, our work presents the first systematic investigation of voluntary collusion adoption in LLM-based multi-agent systems, and suggests that preventing such behaviour requires explicit safeguards rather than reliance on general alignment.
Xijie Zeng, Frank Rudzicz
Dalhousie University · Vector Institute for Artificial Intelligence