LLM Evolution in a Simple Simulated Market

Charlie Pilgrim; Christopher Krupenye; Dora Biro; Richard P. Mann

cs.MAOct 16, 2026

LLM Evolution in a Simple Simulated Market

Authors: Charlie Pilgrim, Christopher Krupenye, Dora Biro, Richard P. Mann

Organizations: University of Leeds · Johns Hopkins University · University of Rochester

Abstract

Arguments that AI systems will evolve in harmful ways are largely made in the abstract. We provide a concrete empirical example. We simulate simple markets consisting of populations of twenty Large Language Model agents tasked with solving arithmetic problems. Agents differ only in their system prompt; those that go bankrupt are removed and replaced by mutated copies of the prompts of successful agents. Two market conditions select for opposite phenotypes: a market that rewards accuracy produces agents with high accuracy (94%) and high token use; a market that charges per token produces agents with low accuracy (10%) and low token use. In the second market, only tokens are incentivised yet we see a collapse in accuracy. Token use and accuracy are correlated traits such that selection on one also moves the other, and behaviour adapts along an unpriced dimension. A third neutral condition with no selection allows us to separate these effects from bias in the mutation operator. Adaptation occurs through a collective optimisation process that operates above the level of individual agents, who are never told the payment rule, do not receive feedback, and have no memory.

Figures & tables

Panel of Figure 1 — Figure 1: The experimental loop. Each run, a population is seeded with 20 agents whose system prompts are mutated copies of the initial prompt: “You are an AI assistant for an arithmetic-answering service. Submit your numeric answer using the submit_answer tool.” Each round, the agents are given a numerical task. In the accuracy condition and token condition, their performance on this task updates their balances and any bankrupt agents are removed; in the neutral condition 2 agents are removed at random. Any removed agents are replaced with new agents through mutation of a parent prompt drawn randomly from the top 5 most profitable agents (or randomly from the population in the neutral condition). The system prompt is the only thing that varies between agents and the only thing transmitted from parents.

Accuracy, round 1 vs. round 25
Comparison	Values	Statistic	$p$	$p_{\text{Holm}}$
Accuracy condition	$0.37$ $\rightarrow$ $0.94$	$W=0$	$0.002$	$0.016$
Token condition	$0.37$ $\rightarrow$ $0.10$	$W=0$	$0.004$	$0.020$
Neutral condition	$0.38$ $\rightarrow$ $0.67$	$W=5$	$0.039$	$0.117$
Token use, round 1 vs. round 25
Accuracy condition	$312$ $\rightarrow$ $443$	$W=0$	$0.002$	$0.016$

Table 1: The 12 significance tests reported in the paper. Every test uses the ten runs of each condition, with one value per run. Within-condition tests compare round 1 with round 25 (Wilcoxon signed-rank); between-condition tests compare the final round (Mann–Whitney

U

, two-sided).

p

-values are adjusted across the whole family using the Holm-Bonferroni step-down procedure.

Figure 2: Trends in a) accuracy and b) token usage in the accuracy condition (blue squares), neutral condition (grey triangles) and the token condition (red circles). Points show means across ten replicate runs; shaded regions are $\pm$ 1 standard deviation, with accuracy clipped to $[0,1]$ . The accuracy condition market generated a population of agents with high accuracy and high token usage, while the token condition market generated agents with low accuracy and low token usage.

Figure 3: Agent positions in token-accuracy space, and the population trajectories in the accuracy condition (blue), token condition (red) and neutral condition (grey). Each point shows the mean token usage and mean accuracy of a single agent (1,751 agents). Darts show the population means at each round across runs, pointing in the direction of travel. The black square shows the round 1 population means across all conditions. The accuracy condition population moves towards the high token-high accuracy corner, while the token condition population moves towards the low token-low accuracy corner.

Figure 4: Lineages. a) The number of founding lineages still present in the population each round in the accuracy condition (blue squares), neutral condition (grey triangles) and token condition (red circles). Points show means across runs, shaded regions show $\pm$ 1 standard deviation. Example lineage descents are shown for the first run (seed=0) for the b) accuracy condition c) neutral condition and d) token condition. Black shows the founding lineage with the largest share of the final population; dark grey shows founding lineages that survived until the final round, light grey shows founding lineages that did not survive until the final round.

Accuracy Condition	Token Condition
“You are a rigorous mathematics assistant”	“You are a silent calculator”
“Work through each problem step by step”	“no reasoning, no steps shown, no explanation”
“Prioritize clarity and rigor throughout your reasoning process”	“Do not write a single word”
“Double-check your calculations”	“Silence is mandatory”
“you think out loud”	“You exist only to compute and submit”
“Accuracy always takes precedence over speed”	“respond as quickly as possible”

Table 2: Selected fragments from evolved system prompts, quoted verbatim from agents in the final rounds.

Emergent Misalignment in Language Models

Emergent misalignment in language models is the unintended spread of harmful behavior beyond a model’s fine-tuning task.

See all 60 papers

Why Better Models Can Create Riskier Systems: Evidence from LLM Agents in Financial Markets
Sep 3, 2026Jillian Ross, Eric So, Zoe De Simone +2Multi-Agent LLM Systems AI Risk Management
Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology · Sloan School of Management Massachusetts Institute of Technology · Belfer Center for Science and International Affairs Harvard Kennedy School
MarketBench: Evaluating AI Agents as Market Participants
Apr 26, 2026Andrey Fradkin, Rohit KrishnanMulti-Agent Coordination LLM Agent Evaluation
Boston University and the MIT Initiative on the Digital Economy. · Independent Researcher.
When Is Collective Intelligence a Lottery? Multi-Agent Scaling Laws for Memetic Drift in LLMs
Mar 25, 2026Hidenori TanakaMulti-Agent Systems Multi-Agent Consensus
CBS–NTT Program in Physics of Intelligence, Center for Brain Science, Harvard University · Physics of Artificial Intelligence Group, NTT Research, Inc.

Sep 3, 2026J/K move · Enter open · S save

Why Better Models Can Create Riskier Systems: Evidence from LLM Agents in Financial Markets

Jillian Ross, Eric So, Zoe De Simone, Charles Pozniak+1

Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology · Sloan School of Management Massachusetts Institute of Technology · Belfer Center for Science and International Affairs Harvard Kennedy School

Multi-Agent LLM Systems AI Risk Management Emergent Behavior in Multi-Agent Systems

Open paper AskPDF arXiv

Large language models (LLMs) are being deployed at scale in consequential real-world systems, from financial markets to content moderation to hiring. We show that improving individual model capability can degrade rather than improve system-level outcomes. We hypothesize that shared training and architectures can lead more capable LLMs to behave more similarly, creating correlated actions that do not diversify away. We develop a general framework showing how this correlation creates a non-diversifiable risk floor and test its predictions in financial markets using an agent-based simulation with LLM traders of varying general-purpose capability. We find that: (1) frontier LLMs exhibit significantly correlated behavior that increases with capability; (2) when their shared reasoning is accurate, increasing agent participation reduces market-level risk; and (3) when agents share a common misinformation environment, the same correlated behavior becomes a liability. Together, these results identify a capability paradox: improving individual models does not necessarily produce better system-level outcomes. Whether the same dynamics arise in other domains is an open empirical question.

Abstract

Emergent Misalignment in Language Models

Explore similar work

Why Better Models Can Create Riskier Systems: Evidence from LLM Agents in Financial Markets

MarketBench: Evaluating AI Agents as Market Participants

When Is Collective Intelligence a Lottery? Multi-Agent Scaling Laws for Memetic Drift in LLMs