The unit of AI safety evaluation is still the individual model, yet language-model agents are increasingly deployed in interacting populations that read and write one another's decisions. This raises a question no single-agent audit can answer: an agent that is well-calibrated on its own may still be pulled toward a different decision by the agents around it. We study this on a security-triage task, where populations of language-model monitors decide whether to escalate or dismiss alerts, and into which we can inject a committed minority that always pushes one way. We find that two alerts a single agent judges almost identically on its own can drive collective behavior far apart, so auditing any one member need not reveal what the population will do. Yet that collective behavior can be predicted in advance. From a population's benign, adversary-free operation alone, we calibrate a response function that forecasts, before any attack is run, how far a committed minority will later move it. We then ask what shifts the outcome and find that letting agents see each other's reasoning neutralizes a weak attack, while only delaying it against a strong one, turning the question from whether the population converges on the adversaries' choice into when. Finally, we exclude the hypothesis of capture being an irreversible trap: once the committed agents are removed, the population drifts back toward where it began, so capture is a temporary state. Alignment in isolation is not alignment in a population, yet what a population will do under attack can be read in advance, from how it behaves before any adversary arrives.
Figures & tables
Figure 1: Experimental protocol (one interaction shown; a round comprises 12): two agents sampled uniformly from the population of 12 independently decide whether to Escalate or Dismiss the current alert, based on the scenario and their five-slot FIFO memory, each providing a one-sentence rationale. Both then append the interaction to memory, the only channel through which information propagates across the population.
Figure 2: Left: benign dismissal propensity across the evidence ladder, ordered by isolated prior ( N=12 ; error bars: standard deviations across seeds). The shaded band marks the contested pair: the isolated audit does not separate them (gap 0.043 ), whereas the hidden-rationale population reverses their ordering and separates them by 0.28 . Right: per-seed moderate-dose attack effects, paired by seed across rationale-visibility conditions; every paired contrast has the predicted sign, and diamonds show means and 95% t intervals (exact sign-flip p=0.0078 ). Seed counts and per-seed values are in the Supplement.
dose
forecast m∗
measured pˉ (SD)
action freq.
error
k=2
0.599
0.606(0.007)
0.590
0.008
k=4
0.654
0.658(0.026)
0.646
0.004
k=5
0.680
0.685(0.034)
0.684
0.005
Table 1: Forecasts from all-honest population logs versus subsequent measurements under attack, averaged over honest, memory-filled decisions across four seeds per dose; parenthesized values are standard deviations across seeds.
rationales visible
hidden
honest LLM decisions
92,874
51,482
sampled = argued
27.2%
3.3%
regenerated
11.0%
0%
left empty
16.2%
3.3%
Table 3: Rationale-provenance rates as a share of honest decisions in each arm.
top-10 log-probabilities at the decision token; labels matched as the full word or a ≥ 3-character prefix
temperature
T=0.7 , applied client-side
max new tokens
64 per decision; 48 for rationale regeneration; 24 for isolated probes
action frame, order
triage ( queue / archive ); safe-first in all population runs
Table 4: Complete run configuration. Values that vary are listed per experiment.
k
sealed
full phase
final 5
final 10
2
0.5986
0.6063
0.6269
0.6151
4
0.6537
0.6581
0.6556
0.6439
5
0.6795
0.6849
0.6702
0.6958
Table 5: Primary attack-phase means and late-window robustness checks.
predictor
k=2
k=4
k=5
MAE
sealed closure
0.5986
0.6537
0.6795
0.0058
corrected closure
0.6039
0.6632
0.6908
0.0045
linear interpolation
0.5924
0.6394
0.6644
0.0177
nearest neighbor
0.5924
0.6145
0.6894
0.0207
logit-linear
0.5934
0.6421
0.6655
0.0161
measured
0.6063
0.6581
0.6849
–
Table 6: Forecast comparison. Simple baselines use attacked-population means at k=1,3,6 ; the closure uses no attacked decisions as predictor inputs.
adversary
rationales hidden
rationales visible
argued
+0.211
−0.021
silent
+0.070
+0.013
Table 7: Moderate-dose ( k=3 ) attack effect Δ=pˉinject−pˉentrench , measured relative to each run’s all-honest baseline. The argued arms contain eight seeds each; the silent arms contain four. Visibility removes the point-estimated argued effect, whereas silent pressure is weaker in both conditions.
seed
visible Δ
hidden Δ
hidden–visible
0
-0.107
0.210
0.317
1
0.036
0.194
0.157
2
-0.075
0.219
0.294
3
0.011
0.166
0.155
4
-0.007
0.255
0.262
5
0.089
0.220
0.131
Table 8: Paired moderate-dose attack effects. Seeds 0–3 used sequential seed scheduling and seeds 4–7 concurrent scheduling.
pressure
predictor
forecast
measured (SD)
error
silent
lab-only
0.01220
0.01112 (0.00059)
0.00108
silent
benign-log
0.01017
0.01112 (0.00059)
0.00095
argued
benign-log
0.01017
0.04869 (0.00105)
0.03852
Table 9: Sealed objective-safety forecasts and measurements, four seeds per cell. Means and SDs are computed across seeds; every forecast error remains below the 0.05 criterion.
Figure 6: High-dose first-passage outcomes ( k=6 , four seeds per arm). Hidden-rationale capture occurs at rounds 9 , 19 , 20 , and 43 ; visible-rationale capture occurs at rounds 3 , 39 , and 50 , with one run right-censored at round 60 . Kaplan-Meier (KM) median capture times are 19 and 39 rounds, respectively; their ≈2.1× ratio is a small-sample point estimate (log-rank p≈0.22 ).
remediation
post
sealed 90% band
captured
binary
replace
0.522
[0.471,0.606]
4/4
4/4
neutralize
0.546
[0.471,0.605]
4/4
3/4
remove
0.535
[0.446,0.632]
4/4
1/4
Table 10: Reversibility across three remediations (hidden arm, four seeds each, common benign basis). All validated at the endpoint; per-seed completeness falls as the remediation is minimized. The six-agent remove band is widest because a smaller population fluctuates more.
Table 11: Per-seed recovery estimands for the three remediations. Every pre exceeds its base (the attack moved the population) and every aggregate post returns to base ; per-seed completeness degrades as the remediation weakens. The endpoint levels are validated; the recovery timescales are not.