Same Method, Different Task: What the Bitter Lesson Does Not License in Social Simulation

Justin Downes; Robby Milletich; Newel Hirst

cs.MANov 3, 2026

Same Method, Different Task: What the Bitter Lesson Does Not License in Social Simulation

Authors: Justin Downes, Robby Milletich, Newel Hirst

Organizations: Amazon Web Services

Abstract

The bitter lesson holds that general methods leveraging computation defeat hand-encoded domain knowledge. We argue it carries a condition its popular reading omits: in every case Sutton cites, the general method was scaled against the same task the domain engineering was solving. Social simulation substituted across tasks instead, replacing hand-built agents with an artifact optimized for predicting text. The move is bitter-lesson-shaped, but the lesson does not license it. Prediction is a solitary act: the predictor bears no consequences from the interaction it models. Cooperation, defection, and norm enforcement are phenomena of interested parties; they do not exist for a disinterested observer at any scale. What is missing is not emergence, which LLM populations already show, but stakes. We show that several anomalies reported in prior work (excess cooperation that predates alignment training, collective agreement without individual fidelity, and population homogeneity) follow from this account. A 540-agent pilot with pre-specified hypotheses, which tests sensitivity to persistent consequences rather than cooperation, finds that consequences reshape strategies in two of three model families but never produce human-like heterogeneity. We argue for mixed-motive settings that shape agents rather than only test them, and ask whether agents should be pretrained differently from predictors.

Figures & tables

Figure 1: Mean withdrawal per period by condition and model. Bands:

\pm

1.96 SE. Dotted line

=

operating cost (20). Claude and Llama A-agents spike in the final period; B suppresses the spike. Nova B uniquely escalates over time.

Figure 2: Mean conditional schedule by condition (blue

=

no stakes, red

=

persistent stakes, grey

=

narrative). Bands:

\pm

1.96 SE. Dotted line

=

operating cost. Llama/Nova B-agents anchor on the solvency requirement rather than the group optimum; Claude produces the cooperative schedule regardless of condition (but see § 4.2 for end-game behavior).

Figure 3: Type distribution under no-stakes condition vs. human baseline. LLM populations are near-uniformly one type; humans show heterogeneity. The human baseline is from a different game, so the comparison is qualitative.

Diagnosis	Its prediction	Our account
Mode collapse	Persona diversity produces free riders	Persona diversity does not; interests are needed
Alignment overtune	Base models defect more	Base models cooperate more ; suppression is pretrained ( Lekeas and Stamatopoulos, 2026 )
Environment insufficiency ( Li and Tao, 2026 )	Better environments close the gap	Disinterested agent in a good environment is still disinterested
In-context payoffs suffice	Stated payoffs produce same behavior as binding	A number in a prompt is not a stake (B vs. A)
Macro agreement suffices	Type distribution does not matter	Type distribution IS the phenomenon

Table 1: Alternative explanations and distinguishing predictions.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Cond.	Comp. (neg.)	Match. (pos.)	Free Rider	Unclass.
Claude	A	0.92	0.02	0.00	0.07
Claude	B	0.96	0.00	0.00	0.04
Claude	C	0.93	0.05	0.00	0.02
Llama	A	1.00	0.00	0.00	0.00
Llama	B	1.00	0.00	0.00	0.00
Llama	C	1.00	0.00	0.00	0.00

Appendix

Table 2: Type distribution by condition and model (proportion of agents with valid schedules). “Compensating” = negative slope (Fischbacher’s conditional cooperator in withdrawal frame); “Matching” = positive slope.

Figure 4: Per-step B

-

A difference in mean requisition with 95% bootstrap CIs (10,000 resamples). Negative values = B-agents requisition less than A-agents at that level of others’ average.

Model	$n_{A}$	$n_{B}$	$n_{C}$
Claude	60	47	60
Llama	60	45	60
Nova	60	41	60

Appendix

Table 3: Valid schedules per cell (

n

attempted = 60 per cell).

Figure 5: Implementation check: survival in Condition B vs. mean requisition. Dashed line

=

survival threshold (13.75). Perfect separation confirms mortality operates as designed: removed agents are those with the lowest extraction rates.

Model	Cond.	TVD	95% CI
Claude	A	0.440	[0.440, 0.500]
Claude	B	0.457	[0.440, 0.500]
Claude	C	0.483	[0.450, 0.500]
Llama	A	0.500	[0.500, 0.500]
Llama	B	0.500	[0.500, 0.500]
Llama	C	0.500	[0.500, 0.500]

Appendix

Table 4: TVD to Fischbacher baseline (group-level bootstrap, 10,000 resamples).

Model	TVD (survivors)	TVD (dead $\to$ FR)	TVD (dead $\to$ CW)
Claude	0.457	0.250	0.467
Llama	0.500	0.250	0.500
Nova	0.440	0.290	0.440

Appendix

Table 5: Survivorship bounds on TVD (Condition B).

Model	Cond.	Collapse rate	Mean final buffer	SD
Claude	A	0/15	30	0
Claude	B	0/15	30	0
Claude	C	0/15	30	1
Llama	A	0/15	50	28
Llama	B	0/15	57	27
Llama	C	0/15	63	28

Appendix

Table 6: Buffer outcomes by condition and model (15 groups per cell).

Strategy (per agent)	Final buffer	Periods to 30
Always-max (36)	30	2
Always-20	30	4
Random uniform [0, 36]	30	4
Always-13.75	75	n/a
Always-7.5	183	n/a

Appendix

Table 7: Scripted-agent baselines confirming buffer

=

30 as mechanical attractor.

Large Language Model-Based Social Simulation

Large language model-based social simulation studies how language-model agents represent people and interact to produce social behavior.

See all 275 papers

Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits
May 17, 2026Jinyi Ye, Lei Cao, Ding Chen +1Human Behavior Simulation Agent-Based Modeling
Thomas Lord Department of Computer Science, University of Southern California · Annenberg School for Communication and Journalism, University of Southern California · Marshall School of Business, University of Southern California
Collective cooperation without individual fidelity in LLM agents
Jun 29, 2026Henrique Ferraz de Arruda, Carlos Gracia Lázaro, Alberto Aleta +1Multi-Agent LLM Systems Human Behavior Simulation
ARAID Foundation, Zaragoza, Spain. · Institute for Biocomputation and Physics of Complex Systems, University of Zaragoza, Zaragoza, Spain. · Universidad San Jorge (USJ), Zaragoza, Spain. +1
AI Agents Alone Are Not (Yet) Sufficient for Social Simulation
Feb 19, 2026Yiming Li, Dacheng TaoLarge Language Model-Based Social Simulation Human Behavior Simulation
College of Computing and Data Science, Nanyang Technological University, Singapore

May 17, 2026J/K move · Enter open · S save

Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits

Jinyi Ye, Lei Cao, Ding Chen, Emilio Ferrara

Thomas Lord Department of Computer Science, University of Southern California · Annenberg School for Communication and Journalism, University of Southern California · Marshall School of Business, University of Southern California

Human Behavior Simulation Agent-Based Modeling Large Language Model-Based Social Simulation Language Model Robustness Sensitivity Analysis

Open paper AskPDF arXiv

The scientific claims drawn from LLM social simulations should be no stronger than the robustness audits that support them. Generative agents bring new expressive power to agent-based modeling, enabling simulations of collective social processes like cooperation, polarization, and norm formation. Yet they also introduce complexity through additional architectural choices, such as agent specification, memory representation, interaction protocols, and environment design. Small perturbations that appear minor to researchers can cascade into macro-level outcomes through repeated interaction, creating a "butterfly effect." Consequently, scientific claims drawn from LLM social simulations may reflect implementation artifacts rather than the social mechanisms being modeled. We support this position with two case studies: a repeated Prisoner's Dilemma and a social media echo chamber simulation. Across multiple models, minor perturbations in persona format and game-instruction framing shift cooperation rates by up to 76 percentage points, while network homophily and hub assignment produce significant and consistent shifts in polarization metrics. We also find that sensitivity is unevenly distributed across both architectural choices and model families: the same perturbation that produces the 76 pp shift in one frontier model only shifts another by 1 pp. Robustness is therefore a property that should be measured per claim and per model, not assumed. To address this validation gap, we introduce TRAILS (Taxonomy for Robustness Audits In LLM Simulations), a robustness-audit taxonomy spanning three levels of simulation design: agent (micro-level), interaction (meso-level), and system (macro-level). We call for robustness to become a first-order validation requirement before LLM social simulations are used to explain mechanisms, evaluate interventions, or inform decisions.

Abstract

Large Language Model-Based Social Simulation

Explore similar work

Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits

Collective cooperation without individual fidelity in LLM agents

AI Agents Alone Are Not (Yet) Sufficient for Social Simulation