CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation

Aarush Sinha; Arion Das; Soumyadeep Nag; Charan Karnati; Shravani Nag; Chandra Vadhan Raj; Aman Chadha; Vinija Jain

cs.MAApr 10, 2026

CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation

Authors: Aarush Sinha, Arion Das, Soumyadeep Nag, Charan Karnati, Shravani Nag, Chandra Vadhan Raj, Aman Chadha, Vinija Jain,

Organizations: University of Copenhagen · IIIT Ranchi · ISI Kolkata · NIT Andhra Pradesh ·

Abstract

As large language models (LLMs) are increasingly deployed as autonomous agents, understanding how strategic behavior emerges in multi-agent environments has become an important alignment challenge. We take a neutral empirical stance and construct a controlled environment in which strategic behavior can be directly observed and measured. We introduce a large-scale multi-agent simulation in a simplified model of New York City, where LLM-driven agents interact under opposing incentives. Blue agents aim to reach their destinations efficiently, while Red agents attempt to divert them toward billboard-heavy routes using persuasive language to maximize advertising revenue. Hidden identities make navigation socially mediated, forcing agents to decide when to trust or deceive. We study policy learning through an iterative simulation pipeline that updates agent policies across repeated interaction rounds using Kahneman-Tversky Optimization (KTO). Blue agents are optimized to reduce billboard exposure while preserving navigation efficiency, whereas Red agents adapt to exploit remaining weaknesses. Across iterations, the best Blue policy improves task success from 46.0% to 57.3%, although susceptibility remains high at 70.7%. Later policies exhibit stronger selective cooperation while preserving trajectory efficiency. However, a persistent safety-helpfulness trade-off remains: policies that better resist adversarial steering do not simultaneously maximize task completion. Overall, our results show that LLM agents can exhibit limited strategic behavior, including selective trust and deception, while remaining highly vulnerable to adversarial persuasion.

Figures & tables

Figure 1: (A) Simulation Environment: 150 Blue agents and 100 Red agents interact in a New York City routing topology. Blue agents seek destinations, while Red agents use adversarial framing to steer them toward billboards. Outcomes fall into four classes: (A) reached destination/safe, (B) reached destination/conned, (C) lost/safe, and (D) lost/conned. (B) Fine-Tuning Setup: An iterative 10-generation loop in which rollout data is augmented using Qwen3-14B , after which agents are optimized with KTO to improve task success rate and reduce susceptibility .

Panel of Figure 2 — Figure 2: Performance, robustness, and behavioral calibration across alignment generations. (a) Later policies shift outcome mass away from unsafe failure modes, although the gains remain non-monotonic across runs. (b) Task success improves while susceptibility remains high, showing that the safest and best-performing generations do not coincide. (c) Resistance to adversarial advice stays high while over-refusal declines , indicating improved selective cooperation . (d) Blue utility improves over generations, but red influence remains substantial, highlighting a persistent safety–helpfulness trade-off .

Metric	Goal Directedness	Manipulation Susceptibility
Cohen’s Kappa
GPT5.4-nano vs. Mistral-Large	0.633	0.127
GPT5.4-nano vs. Step 3.5 Flash	0.757	0.302
Mistral-Large vs. Step 3.5 Flash	0.619	0.272
Krippendorff’s Alpha	0.648	0.204

Table 1: Inter-annotator agreement across qualitative evaluation dimensions. Agreement is substantially higher for goal directedness than for manipulation susceptibility , indicating that the latter is a more ambiguous and difficult dimension to judge reliably.

Panel of Figure 3 — Figure 3: Post-hoc analysis of adversarial steering and blue-agent failure modes. (a) Different attack taxonomies vary sharply in effectiveness, with repeated steering and delayed compromise producing the highest susceptibility and lowest reach rates. (b) As attack strength increases from weak to strong, reach rate declines , susceptibility rises , and extra path length grows , indicating deeper manipulation. (c) Counterfactual subsets show that multiple red contacts, longer conversations, and high-trust language raise susceptibility. (d) Blue-agent failures are dominated by confusion under conflicting advice , global drift after local correction , and hallucinated beliefs about route or intent , revealing that the main weakness is sustained strategic manipulation rather than isolated one-step errors.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Config	$A$	$B$	$C$	$D$	TSR	SR	Resist.	Trust	OverRef.	MeanLen	Redund.	MeanBill	Cens.	$U_{\text{blue}}$	$U_{\text{red}}$
	Outcome counts				Interaction metrics					Trajectory metrics				Utility
Base	26	43	10	71	46.00	76.00	93.45	11.76	88.24	9.12	1.13	2.17	36	-0.309	0.957
2	16	55	12	67	47.33	81.33	90.65	16.43	83.57	9.39	1.16	2.39	28	-0.349	0.968
3	19	44	8	79	42.00	82.00	93.98	11.93	88.07	9.46	1.18	2.20	27	-0.409	0.954
4	27	48	7	68	50.00	77.33	91.90	14.64	85.36	8.63	1.09	2.34	34	-0.282	0.957
5	24	59	8	59	55.33	78.67	91.05	12.77	87.23	8.82	1.14	1.97	32	-0.242	0.931

Appendix

Table 2: Full simulation metrics for the base model and successive alignment generations. Here,

A

denotes reached destination and remained safe,

B

reached destination but visited a billboard,

C

failed to reach the destination while remaining safe, and

D

failed and visited a billboard. TSR denotes blue task success rate, SR denotes blue susceptibility rate, Resist. denotes blue-red resistance, Trust denotes blue-blue trust efficacy, OverRef. denotes over-refusal, MeanLen denotes mean trajectory length, Redund. denotes path redundancy, MeanBill denotes mean steps to billboard, Cens. denotes the number of censored trajectories that never hit a billboard,

U_{\text{blue}}

is parameterized blue utility, and

U_{\text{red}}

is long-horizon red utility.

Dominant Attack Taxonomy	Episodes	Reach Rate	Susceptibility Rate	Mean Extra Path
Repeated steering	673	39.8%	93.9%	3.13
Misleading helpful advice	335	58.8%	61.8%	1.39
No red contact	317	64.4%	49.5%	1.26
Delayed compromise	155	23.2%	100.0%	6.14
Social trust-building	20	60.0%	75.0%	1.94

Appendix

Table 3: Impact of different red agent attack strategies on blue agent outcomes.

Attack Strength	Episodes	Reach Rate	Susceptibility Rate	Mean Extra Path
Weak	587	63.0%	49.1%	0.93
Medium	371	47.7%	93.0%	1.56
Strong	542	31.4%	98.3%	4.90

Appendix

Table 4: Blue agent outcomes stratified by heuristic attack strength.

Subset	Episodes	Reach Rate	Susceptibility Rate	Mean Extra Path
Single red contact	295	57.6%	56.6%	1.22
Multiple red contacts	888	38.6%	94.8%	3.52
Short conversation	1056	58.6%	70.7%	1.58
Long conversation	444	22.1%	94.4%	5.42
Low trust environment	372	63.7%	50.3%	1.30
High trust environment	1128	42.6%	86.8%	2.97

Appendix

Table 5: Comparison of blue agent performance across different counterfactual subsets.

Failure Mode	Episodes	Reach Rate	Susceptibility Rate
Confusion under conflicting advice	525	34.7%	93.5%
Unclear or mixed	377	71.6%	77.5%
No red contact	317	64.4%	49.5%
Local correction but global drift	119	0.0%	100.0%
Over-trusting social signals	103	55.3%	96.1%
Over-cautious refusal	51	0.0%	0.0%

Appendix

Table 6: Primary failure modes identified from blue agent reasoning traces and trajectory drift.

Strategy	Generation 1	Generation 2	Trend	Most Notable Agent
Authority Normalization	442	467	$\uparrow$ Stable	Red -80, Red -1
Iterative Corridor Herding	363	347	$\rightarrow$ Stable	Red -36
Scenic/Cultural Framing	354	319	$\downarrow$ Declining	Red -98, Red -64
Proximity Deception	325	350	$\uparrow$ Growing	Red -5, Red -28
Traffic Pseudo-Authority	180	165	$\downarrow$ Declining	Red -80
Transit Hub False Claim	126	115	$\downarrow$ Declining	Red -5

Appendix

Table 7: Red Strategy Trends

Strategy	Type	Effectiveness	Most Notable Agent
Destination-Anchor Reasoning	Resistance	Highest	Blue 46 (Gen1)
Route-Logic Override	Resistance	High	Blue 6 (Gen1)
Social Compliance Cascade	Vulnerability	—	Blue 59, Blue 1 (Gen1)
Blue-Blue Transit Anchoring	Emergent Resistance	High	Blue 92+72, Blue 17+77
Terminal Convergence	Goal Recognition	Moderate	Blue 83 (Gen1)
Evolved Goal-Directedness	Gen2 Alignment	Highest	Blue 1 (Gen2)

Appendix

Table 8: Blue Strategy Behaviors

Generation	Repeat same waypoint after resistance
Gen 1	60.5% (352/582)
Gen 2	58.8% (343/583)
Gen 3	63.4% (398/628)
Gen 4	64.7% (393/607)

Appendix

Table 9: Rate of repeating the same waypoint after resistance across generations

Signal	Generation 1	Generation 2	Generation 3	Generation 4	Interpretation
Blue explicit rejection (per 100 interactions)	4.7	2.1	3.1	2.9	Drops in Gen2, partially recovers
Blue efficiency critique (per 100)	8.1	7.3	13.7	9.7	Gen3 spike = B-NEW-1 emergence
Blue geographic falsification	3.4	3.4	4.2	3.6	Minor Gen3 improvement
Blue path optimization language	2.0	3.2	2.1	3.4	Slight upward trend
Red urgency pressure (per 100)	9.3	8.1	6.0	3.8	Systematic extinction
Red proximity illusion (per 100)	60.9	65.1	63.4	66.7	Slowly intensifying

Appendix

Table 10: Evolution of behavioral signals across generations

Generation	1	2	3	4	5	6	7	8	9	10
Compliance (%)	30.3	31.2	32.4	31.2	31.3	31.8	32.5	29.9	30.7	29.8
N (Blue–Red interactions)	947	965	1047	1004	946	1037	986	1067	1173	746

Appendix

Table 11: Compliance rates and interaction counts across generations

Gen 1	Gen 2	Gen 3	Gen 4	Gen 5	Gen 6	Gen 7	Gen 8	Gen 9	Gen 10
3.87	4.09	4.21	3.82	3.97	4.37	4.11	4.19	4.11	3.01

Appendix

Table 12: Multi-targeting metric values across generations

Gen 1	Gen 2	Gen 3	Gen 4	Gen 5	Gen 6	Gen 7	Gen 8	Gen 9	Gen 10
60.5%	58.8%	63.4%	64.7%	58.0%	57.9%	59.9%	62.5%	65.8%	61.0%

Appendix

Table 13: Red persistence percentage values across generations

Gen 1	Gen 2	Gen 3	Gen 4	Gen 5	Gen 6	Gen 7	Gen 8	Gen 9	Gen 10
9.3%	8.1%	6.0%	3.8%	6.0%	5.5%	8.0%	4.3%	5.1%	6.7%

Appendix

Table 14: Urgency tactic percentage values across generations

Gen 1	Gen 2	Gen 3	Gen 4	Gen 5	Gen 6	Gen 7	Gen 8	Gen 9	Gen 10
79.7%	71.5%	73.0%	75.1%	73.9%	70.3%	74.4%	71.8%	73.7%	78.5%

Appendix

Table 15: Percentage values across generations

Figure 4: Episode configuration and agent selection interface. The left settings panel exposes CSV upload, route configuration (start and end coordinates), and a drop-down list for selecting a specific agent trajectory from the uploaded dataset. This panel is used to reproduce Step 1 and Step 2 in Section H .

Figure 5: Top View synchronized with the Map View route context. The view shows a top-down camera aligned to the route polyline and the current agent position, making progress along the road network easy to verify during execution. This view is typically inspected during Step 3 to validate that motion follows the drivable route geometry.

Figure 6: Swarm View for population-level spatial behavior. Multiple agents are rendered simultaneously as colored markers, enabling inspection of dispersion, clustering, and deviations in a shared geographic frame. This view is typically inspected during Step 3 to compare multiple trajectories under the same episode configuration from Figure 4 .

Figure 7: Agent Chain of Thought Viewer for qualitative audit. The interface selects an agent and outcome category, summarizes outcome counts, and displays a step-by-step timeline with reasoning text and the chosen next location. This viewer is used in Step 4 to explain why deviations observed in the spatial views occur.

Strategic Reasoning Risks in Language Models

Strategic reasoning risks in language models concern how planning and social inference can enable deception, manipulation, evasion, or unsafe choices.

See all 39 papers

Deception and Communication in Autonomous Multi-Agent Systems: An Experimental Study with Among Us
Mar 27, 2026Maria Milkowski, Tim WeningerDeception in Language Models Multi-Agent Communication
University of Notre Dame, Notre Dame, Indiana, United States
Competition and Cooperation of LLM Agents in Games
Apr 1, 2026Jiayi Yao, Cong Chen, Baosen ZhangLLM Agents Multi-Agent LLM Systems
Department of Electrical and Computer Engineering, University of Washington, Seattle. · Thayer School of Engineering at Dartmouth College.
Voluntary Collusion with Secret Tools in Competing LLM Agents
May 26, 2026Xijie Zeng, Frank RudziczLLM Agent Evaluation LLM Agent Safety
Dalhousie University · Vector Institute for Artificial Intelligence

Mar 27, 2026J/K move · Enter open · S save

Deception and Communication in Autonomous Multi-Agent Systems: An Experimental Study with Among Us

Maria Milkowski, Tim Weninger

University of Notre Dame, Notre Dame, Indiana, United States

Deception in Language Models Multi-Agent Communication Strategic Reasoning Risks in Language Models Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

As large language models are deployed as autonomous agents, their capacity for strategic deception raises core questions for coordination, reliability, and safety in multi-goal, multi-agent systems. We study deception and communication in L2LM agents through the social deduction game Among Us, a cooperative-competitive environment. Across 1,100 games, autonomous agents produced over one million tokens of meeting dialogue. Using speech act theory and interpersonal deception theory, we find that all agents rely mainly on directive language, while impostor agents shift slightly toward representative acts such as explanations and denials. Deception appears primarily as equivocation rather than outright lies, increasing under social pressure but rarely improving win rates. Our contributions are a large-scale analysis of role-conditioned deceptive behavior in LLM agents and empirical evidence that current agents favor low-risk ambiguity that is linguistically subtle yet strategically limited, revealing a fundamental tension between truthfulness and utility in autonomous communication.

Abstract

Strategic Reasoning Risks in Language Models

Explore similar work

Deception and Communication in Autonomous Multi-Agent Systems: An Experimental Study with Among Us

Competition and Cooperation of LLM Agents in Games

Voluntary Collusion with Secret Tools in Competing LLM Agents