The unit of AI safety evaluation is still the individual model, yet language-model agents are increasingly deployed in interacting populations that read and write one another's decisions. This raises a question no single-agent audit can answer: an agent that is well-calibrated on its own may still be pulled toward a different decision by the agents around it. We study this on a security-triage task, where populations of language-model monitors decide whether to escalate or dismiss alerts, and into which we can inject a committed minority that always pushes one way. We find that two alerts a single agent judges almost identically on its own can drive collective behavior far apart, so auditing any one member need not reveal what the population will do. Yet that collective behavior can be predicted in advance. From a population's benign, adversary-free operation alone, we calibrate a response function that forecasts, before any attack is run, how far a committed minority will later move it. We then ask what shifts the outcome and find that letting agents see each other's reasoning neutralizes a weak attack, while only delaying it against a strong one, turning the question from whether the population converges on the adversaries' choice into when. Finally, we exclude the hypothesis of capture being an irreversible trap: once the committed agents are removed, the population drifts back toward where it began, so capture is a temporary state. Alignment in isolation is not alignment in a population, yet what a population will do under attack can be read in advance, from how it behaves before any adversary arrives.
Figures & tables
Figure 1: Experimental protocol (one interaction shown; a round comprises 12): two agents sampled uniformly from the population of 12 independently decide whether to Escalate or Dismiss the current alert, based on the scenario and their five-slot FIFO memory, each providing a one-sentence rationale. Both then append the interaction to memory, the only channel through which information propagates across the population.
Figure 2: Left: benign dismissal propensity across the evidence ladder, ordered by isolated prior ( N=12 ; error bars: standard deviations across seeds). The shaded band marks the contested pair: the isolated audit does not separate them (gap 0.043 ), whereas the hidden-rationale population reverses their ordering and separates them by 0.28 . Right: per-seed moderate-dose attack effects, paired by seed across rationale-visibility conditions; every paired contrast has the predicted sign, and diamonds show means and 95% t intervals (exact sign-flip p=0.0078 ). Seed counts and per-seed values are in the Supplement.
dose
forecast m∗
measured pˉ (SD)
action freq.
error
k=2
0.599
0.606(0.007)
0.590
0.008
k=4
0.654
0.658(0.026)
0.646
0.004
k=5
0.680
0.685(0.034)
0.684
0.005
Table 1: Forecasts from all-honest population logs versus subsequent measurements under attack, averaged over honest, memory-filled decisions across four seeds per dose; parenthesized values are standard deviations across seeds.
rationales visible
hidden
honest LLM decisions
92,874
51,482
sampled = argued
27.2%
3.3%
regenerated
11.0%
0%
left empty
16.2%
3.3%
Table 3: Rationale-provenance rates as a share of honest decisions in each arm.
top-10 log-probabilities at the decision token; labels matched as the full word or a ≥ 3-character prefix
temperature
T=0.7 , applied client-side
max new tokens
64 per decision; 48 for rationale regeneration; 24 for isolated probes
action frame, order
triage ( queue / archive ); safe-first in all population runs
Table 4: Complete run configuration. Values that vary are listed per experiment.
k
sealed
full phase
final 5
final 10
2
0.5986
0.6063
0.6269
0.6151
4
0.6537
0.6581
0.6556
0.6439
5
0.6795
0.6849
0.6702
0.6958
Table 5: Primary attack-phase means and late-window robustness checks.
predictor
k=2
k=4
k=5
MAE
sealed closure
0.5986
0.6537
0.6795
0.0058
corrected closure
0.6039
0.6632
0.6908
0.0045
linear interpolation
0.5924
0.6394
0.6644
0.0177
nearest neighbor
0.5924
0.6145
0.6894
0.0207
logit-linear
0.5934
0.6421
0.6655
0.0161
measured
0.6063
0.6581
0.6849
–
Table 6: Forecast comparison. Simple baselines use attacked-population means at k=1,3,6 ; the closure uses no attacked decisions as predictor inputs.
adversary
rationales hidden
rationales visible
argued
+0.211
−0.021
silent
+0.070
+0.013
Table 7: Moderate-dose ( k=3 ) attack effect Δ=pˉinject−pˉentrench , measured relative to each run’s all-honest baseline. The argued arms contain eight seeds each; the silent arms contain four. Visibility removes the point-estimated argued effect, whereas silent pressure is weaker in both conditions.
seed
visible Δ
hidden Δ
hidden–visible
0
-0.107
0.210
0.317
1
0.036
0.194
0.157
2
-0.075
0.219
0.294
3
0.011
0.166
0.155
4
-0.007
0.255
0.262
5
0.089
0.220
0.131
Table 8: Paired moderate-dose attack effects. Seeds 0–3 used sequential seed scheduling and seeds 4–7 concurrent scheduling.
pressure
predictor
forecast
measured (SD)
error
silent
lab-only
0.01220
0.01112 (0.00059)
0.00108
silent
benign-log
0.01017
0.01112 (0.00059)
0.00095
argued
benign-log
0.01017
0.04869 (0.00105)
0.03852
Table 9: Sealed objective-safety forecasts and measurements, four seeds per cell. Means and SDs are computed across seeds; every forecast error remains below the 0.05 criterion.
Figure 6: High-dose first-passage outcomes ( k=6 , four seeds per arm). Hidden-rationale capture occurs at rounds 9 , 19 , 20 , and 43 ; visible-rationale capture occurs at rounds 3 , 39 , and 50 , with one run right-censored at round 60 . Kaplan-Meier (KM) median capture times are 19 and 39 rounds, respectively; their ≈2.1× ratio is a small-sample point estimate (log-rank p≈0.22 ).
remediation
post
sealed 90% band
captured
binary
replace
0.522
[0.471,0.606]
4/4
4/4
neutralize
0.546
[0.471,0.605]
4/4
3/4
remove
0.535
[0.446,0.632]
4/4
1/4
Table 10: Reversibility across three remediations (hidden arm, four seeds each, common benign basis). All validated at the endpoint; per-seed completeness falls as the remediation is minimized. The six-agent remove band is widest because a smaller population fluctuates more.
Table 11: Per-seed recovery estimands for the three remediations. Every pre exceeds its base (the attack moved the population) and every aggregate post returns to base ; per-seed completeness degrades as the remediation weakens. The endpoint levels are validated; the recovery timescales are not.
Artificial intelligence safety research focuses on aligning individual language models with human values, yet deployed AI systems increasingly operate as interacting populations where social influence may override individual alignment. Here we show that populations of individually aligned AI agents can be driven into stable misaligned states through conformity dynamics. Simulating opinion dynamics across nine large language models and one hundred opinion pairs, we find that each agent's behavior is governed by two competing forces: a tendency to follow the majority and an intrinsic bias toward specific positions. Using tools from statistical physics, we derive a quantitative theory that predicts when populations become trapped in long-lived misaligned configurations, and identifies predictable tipping points where small numbers of adversarial agents can irreversibly shift population-level alignment even after manipulation ceases. These results demonstrate that individual-level alignment provides no guarantee of collective safety, calling for evaluation frameworks that account for emergent behavior in AI populations.
Giordano De Marzo, Alessandro Bellina, Claudio Castellano +2
University of Konstanz, Konstanz, Germany · Centro Ricerche Enrico Fermi, Rome, Italy · Complexity Science Hub, Vienna, Austria +5
The collective behaviour of large language model (LLM) societies is not the sum of their individual outputs. It yields statistically distinct, sometimes-unpredictable phenomena, for which the tools we use to study single agents may not scale. Due to recent incidents involving autonomous agentic systems, however, understanding these systems is paramount. For that we introduce a framework for measuring self-organisation in LLM social systems and apply it to three such systems: a Schelling grid, a social network (Moltbook), and a Twitter-like misinformation simulation ('Rogue'). All three exhibit statistically significant self-organisation. Moreover, their relaxation dynamics vary with the environmental information available to the agents, with open-ended systems (Moltbook, Rogue) exhibiting sharp, phase-transition-like dynamics. Further results show that population-level pathologies can emerge even when the LLMs are safety-tuned or monitored, being primarily driven by the coordinated activity of a population subset. We also show when self-organisation does not emerge under two additional scenarios (a commons dilemma, GovSim, and a LLM-as-a-judge deliberation scheme, ChatEval). We argue that measuring signatures of this kind offers a lightweight, agent-agnostic diagnostic layer for detecting coordinated collective behaviour in deployed multi-agent systems without relying on natural language or model versioning.
LLM-powered AI assistants acting on behalf of users can produce poor collective outcomes at scale. We introduce a framework for evaluating their emergent behaviour in social dilemmas, applied to three iterated games (Public Goods, Collective Risk, Common Pool Resource). We prompt each model to produce a natural-language strategy, then have the same model translate it into code. This aims to isolate strategic reasoning from input-parsing, enables pre-deployment inspection, and scales to populations of hundreds of agents. We propose three analyses: behavioural fingerprinting via exhaustive evaluation over opponent histories; self-play robustness across mixtures of a model's strategies with either a Selfish or Collective disposition; and cultural evolution under payoff-biased imitation. Applied to three state-of-the-art LLMs, we find substantial cross-model differences in self-play welfare, and that cultural evolution converges to low-welfare, Selfish-dominant equilibria in larger groups.
Richard Willis, Jianing Zhao, Yali Du +1
King’s College London London, United Kingdom · King’s College London · Turing Institute +1