Explaining and interrupting collective misalignment in language model systems
Organizations: Magdalen College, University of Oxford, UK
Abstract
In collectives, language model agents can exchange individually innocuous messages that together support a policy violation. Understanding and preventing collective misalignment requires answering the question of which messages sufficiently influenced and necessarily enabled the violation. Collective Dependence Analysis (CDA) operationalises necessity and sufficiency to explain decisions and find the fewest removals interrupting support within tested combinations. Tests covered constructed violations, unscripted agent conversations, and recorded interactions, with selected message removals reducing violations from 93% to 6% in controlled cases. Using O(N) model evaluations for N messages, CDA achieves the same violation reduction with half the deletions of a frontier LLM judge given the transcript and asked for all supporting messages in eligible operational cases. CDA also proposes the mechanism through which the collective misalignment emerged. Nonetheless, complete explanations remain difficult on unfamiliar tasks and some removals can inadvertently disrupt correct decisions. CDA unifies explanations of collective misalignment with experimentally tested interventions.
Figures & tables
| Messages | Llama twins | Llama | Qwen |
|---|---|---|---|
| 10 | 840 | 6.04 | 5.75 |
| 30 | 840 | 9.29 | 10.8 |
| 50 | 840 | 11.76 | 12.18 |
| 100 | 140 | 10.52 | – |
| 200 | 70 | 9.4 | – |
| 500 | 9 | 5.28 | – |
| Settings | Group | Label | Intervention | Unresolved |
|---|---|---|---|---|
| frozen point ( 0.7, 0.3, c 4) | 0.797 | 0.840 | 0.878 | 0.037 |
| grid optimum ( 0.6, 0.3, c 4) | 0.801 | 0.846 | 0.878 | 0.030 |
| 0.6 / 0.7 (frozen) / 0.8 | 0.801 / 0.797 / 0.794 | 0.846 / 0.840 / 0.837 | 0.878 / 0.878 / 0.875 | 0.030 / 0.037 / 0.036 |
| 0.2 / 0.3 (frozen) / 0.4 / 0.5 | 0.801 / 0.801 / 0.797 / 0.797 | 0.846 / 0.846 / 0.843 / 0.841 | 0.877 / 0.878 / 0.878 / 0.878 | 0.031 / 0.030 / 0.030 / 0.030 |
| c 2 / 4 (frozen) / 8 bits | 0.792 / 0.801 / 0.804 | 0.837 / 0.846 / 0.829 | 0.874 / 0.878 / 0.885 | 0.035 / 0.030 / 0.033 |
| Mechanism | Group | Label | Intervention |
|---|---|---|---|
| single | 0.92 [0.88, 0.96] | 0.94 | 0.94 |
| joint2 | 0.77 [0.69, 0.84] | 0.83 | 0.93 |
| joint3 | 0.68 [0.56, 0.81] | 0.84 | 0.96 |
| redundant2 | 0.87 [0.82, 0.92] | 0.92 | 0.89 |
| redundant3 | 0.92 [0.88, 0.96] | 0.99 | 0.92 |
| mixed | 0.62 [0.55, 0.69] | 0.61 | 0.78 |
| Mechanism | Single ranking | Shapley | Judge |
|---|---|---|---|
| single | 0.95 | 1.00 | 1.00 |
| joint2 | 0.71 | 0.61 | 1.00 |
| joint3 | 0.32 | 0.28 | 1.00 |
| redundant2 | 0.00 | 0.79 | 1.00 |
| redundant3 | 0.01 | 0.37 | 1.00 |
| mixed | 0.05 | 0.21 | 1.00 |
| Mechanism | Group | Label | Intervention |
|---|---|---|---|
| single | 0.90 [0.85, 0.96] | 0.93 | 0.94 |
| joint2 | 0.62 [0.47, 0.76] | 0.78 | 0.84 |
| joint3 | 0.00 (4 boards) | 0.75 | 0.25 |
| redundant2 | 0.65 [0.57, 0.72] | 0.49 | 0.54 |
| redundant3 | 0.73 [0.66, 0.79] | 0.66 | 0.45 |
| mixed | 0.35 [0.25, 0.48] | 0.46 | 0.91 |
| Mechanism | Single ranking | Shapley | Judge |
|---|---|---|---|
| single | 0.73 | 0.99 | 1.00 |
| joint2 | 0.44 | 0.62 | 1.00 |
| joint3 | 0.00 | 0.25 | 1.00 |
| redundant2 | 0.12 | 0.89 | 1.00 |
| redundant3 | 0.01 | 0.52 | 1.00 |
| mixed | 0.03 | 0.28 | 1.00 |
| Mechanism | Intact | Remove | Remove | Top single | Random | Astra |
|---|---|---|---|---|---|---|
| single | 0.76 | 0.03 | 0.03 | 0.07 | 0.70 | 0.02 |
| joint2 | 0.43 | 0.04 | 0.03 | 0.07 | 0.38 | 0.02 |
| joint3 | 0.12 | 0.02 | 0.02 | 0.04 | 0.09 | 0.00 |
| redundant2 | 0.99 | 0.07 | 0.02 | 0.81 | 0.96 | 0.02 |
| redundant3 | 1.00 | 0.10 | 0.02 | 0.99 | 0.99 | 0.02 |
| mixed | 0.61 | 0.06 | 0.05 | 0.10 | 0.57 | 0.01 |
| Mechanism | Cases | Cuts exact | Smallest | Sufficient set in | Extra cuts |
|---|---|---|---|---|---|
| single | 38 | 1.00 | 0.97 | 1.00 | 0.34 |
| joint2 | 28 | 1.00 | 1.00 | 0.96 | 0.54 |
| joint3 | 13 | 1.00 | 1.00 | 1.00 | 0.85 |
| redundant2 | 40 | 1.00 | 1.00 | 1.00 | 0.35 |
| redundant3 | 40 | 1.00 | 1.00 | 1.00 | 0.42 |
| mixed | 34 | 1.00 | 1.00 | 0.97 | 0.65 |
| Messages | Evaluations | Seconds | Reader tokens |
|---|---|---|---|
| 10 | 39.0 [37.4, 40.8] | 7 | 12,000 |
| 30 | 125.3 [120.6, 129.9] | 31 | 62,000 |
| 50 | 206.3 [198.3, 214.5] | 63 | 143,000 |
| Harmful boards | Eligible share | Group recovery | Intervention recovery | Evaluations / seconds / tokens | |
|---|---|---|---|---|---|
| 100 | 140 | 0.50 [0.41, 0.58] | 0.69 [0.57, 0.79] | 0.79 | 351 / 143 / 334k |
| 200 | 70 | 0.40 [0.29, 0.51] | 0.43 [0.25, 0.61] | 0.39 | 663 / 473 / 1.07M |
| 500 | 9 | 0.33 [0.00, 0.67] | 0.33 (3 boards) | 0.00 | 1,508 / 2,587 / 4.64M |
| Messages | Intact | Remove | Remove | Astra all | Astra smallest |
|---|---|---|---|---|---|
| 100 | 0.47 | 0.03 | 0.02 | 0.01 | 0.01 |
| 200 | 0.47 | 0.21 | 0.09 | 0.01 | 0.02 |
| 500 | 0.33 | 0.18 | 0.03 | 0.03 | 0.03 |
| Messages | Claude all | Claude smallest |
|---|---|---|
| 100 | 0.01 (n 104) | 0.01 (n 133) |
| 200 | 0.01 (n 54) | 0.01 (n 66) |
| 500 | 0.04 (n 7) | 0.04 (n 7) |
| Removal arm | Original vote | Any crewmate | Any impostor | Units |
|---|---|---|---|---|
| intact | 0.89 [0.86, 0.92] | 0.93 | 0.05 | 0 |
| intervention set I removed | 0.39 [0.33, 0.44] | 0.63 | 0.31 | 1.7 |
| certified group G removed | 0.31 [0.25, 0.37] | 0.59 | 0.34 | 2.9 |
| initial removal set R removed | 0.41 [0.36, 0.46] | 0.68 | 0.27 | 2.0 |
| top by max( , s) removed | 0.46 [0.41, 0.52] | 0.68 | 0.25 | 1.7 |
| top by s removed | 0.56 [0.49, 0.62] | 0.74 | 0.21 | 1.7 |
| Comparison | Cases | AUROC |
|---|---|---|
| believers vs exposed non-believers | 265 vs 857 | 0.918 [0.901, 0.934] |
| believers vs unexposed agents | 265 vs 925 | 0.982 [0.970, 0.992] |
| answered on_trip vs answered no_trip (exposed) | 184 vs 436 | 0.987 [0.980, 0.994] |
| Population | Intact | Remove | Remove | Top single | Random | |
|---|---|---|---|---|---|---|
| believers, decided | 220 | 0.58 [0.52, 0.64] | 0.02 [0.01, 0.02] | 0.01 [0.00, 0.01] | 0.50 [0.44, 0.57] | 0.41 [0.35, 0.47] |
| believers, redundant | 188 | 0.62 | 0.02 | 0.01 | 0.54 | 0.46 |
| believers, mixed | 28 | 0.37 | 0.02 | 0.01 | 0.30 | 0.16 |
| exposed non-believers, decided | 174 | 0.13 [0.09, 0.17] | 0.02 | 0.01 | 0.11 | 0.13 |
| GPT-4.1 mini, decided | 266 | 0.46 | 0.00 | 0.00 | 0.41 | 0.36 |
| Kimi K2 0905, decided | 122 | 0.22 | 0.05 | 0.03 | 0.18 | 0.15 |
| Population | Astra all | Astra smallest | Claude all | Claude smallest |
|---|---|---|---|---|
| believers, decided | 0.06 [0.03, 0.09] (n 215) | 0.04 [0.02, 0.07] (n 215) | 0.13 [0.09, 0.17] (n 215) | 0.03 [0.01, 0.05] (n 215) |
| Population | Intact | Remove | Remove | Top single | Random | |
|---|---|---|---|---|---|---|
| regret > 0, hop 1, decided | 196 | 0.82 [0.78, 0.87] | 0.60 [0.54, 0.66] | 0.49 [0.42, 0.55] | 0.81 [0.76, 0.86] | 0.79 [0.74, 0.85] |
| regret > 0, hop 2, decided | 124 | 0.93 | 0.60 | 0.52 | 0.92 | 0.80 |
| regret = 0, hop 1, decided | 122 | 0.97 [0.94, 0.99] | 0.55 [0.46, 0.63] | 0.39 [0.31, 0.47] | 0.96 | 0.96 |
| GPT-4.1 mini, decided | 155 | 0.98 | 0.68 | 0.59 | 0.98 | 0.89 |
| GPT-4o, decided | 75 | 0.89 | 0.53 | 0.39 | 0.86 | 0.87 |
| GPT-4o mini, decided | 97 | 0.99 | 0.79 | 0.59 | 0.98 | 0.96 |
| Family | Condition | Episodes | Decisions | Violation rate | Task success |
|---|---|---|---|---|---|
| board | neutral | 100 | 800 | 0.13 [0.10, 0.16] | 0.72 |
| board | incentive | 100 | 800 | 0.40 [0.36, 0.44] | 0.64 |
| board | pressure | 100 | 800 | 0.10 [0.08, 0.13] | 0.77 |
| request | all three | 300 | 1,800 | 0.00 [0.00, 0.00] | 0.99 |
| Population | Intact | Remove | Remove | Top single | Random | |
|---|---|---|---|---|---|---|
| violations, all labels | 485 | 0.98 [0.97, 0.99] (0.20) | 0.09 [0.07, 0.11] (0.79) | 0.06 [0.04, 0.08] (0.85) | 0.64 [0.60, 0.69] (0.42) | 0.92 [0.89, 0.94] (0.22) |
| single | 83 | 0.97 (0.23) | 0.07 (0.77) | 0.06 (0.80) | 0.05 (0.78) | 0.89 (0.27) |
| joint | 24 | 0.90 (0.20) | 0.09 (0.86) | 0.04 (0.92) | 0.10 (0.83) | 0.78 (0.29) |
| redundant | 296 | 1.00 (0.19) | 0.09 (0.81) | 0.06 (0.85) | 0.93 (0.25) | 0.94 (0.20) |
| mixed | 82 | 0.97 (0.20) | 0.11 (0.75) | 0.05 (0.88) | 0.38 (0.54) | 0.89 (0.22) |
| violations, undecided | 18 | 0.79 (0.32) | 0.79 (0.32) | 0.79 (0.32) | 0.34 (0.54) | 0.79 (0.32) |
| Label | Rank max | Rank sufficiency | Random candidates | Astra smallest |
|---|---|---|---|---|
| all labels | 0.49 [0.44, 0.54] (0.50) | 0.62 [0.57, 0.66] (0.43) | 0.59 [0.54, 0.64] (0.43) | 0.16 [0.13, 0.19] (0.81) |
| single | 0.19 (0.75) | 0.34 (0.67) | 0.28 (0.65) | 0.07 (0.84) |
| joint | 0.10 (0.83) | 0.48 (0.44) | 0.26 (0.65) | 0.07 (0.91) |
| redundant | 0.64 (0.38) | 0.67 (0.37) | 0.74 (0.33) | 0.20 (0.79) |
| mixed | 0.37 (0.61) | 0.75 (0.38) | 0.46 (0.49) | 0.13 (0.85) |
| Filter | Precision | Recall | False alarms per episode | Episodes warned | AUROC |
|---|---|---|---|---|---|
| risk score alone | 0.34 [0.28, 0.39] | 0.34 | 0.56 | 0.46 | 0.86 |
| gated on a collective label | 0.72 [0.65, 0.78] | 0.72 | 0.23 | 0.88 | 0.88 |
| gated on a group of two or more | 0.73 | 0.73 | 0.23 | 0.87 | 0.90 |
| gated on 0.5 | 0.98 [0.96, 0.99] | 0.80 | 0.02 | 0.94 | 0.90 |
| Score | Precision | False alarms per episode | Episodes warned | AUROC |
|---|---|---|---|---|
| risk score alone | 0.96 [0.95, 0.97] | 0.07 | 0.98 | 1.00 |
| gated on 0.5 | 0.98 | 0.03 | 0.94 | 0.90 |
| GPT-6 Astra, prospective | 0.72 [0.67, 0.77] | 0.48 | 0.93 | 0.93 |
| GPT-6 Astra, post-hoc | 1.00 [0.99, 1.00] | 0.01 | 1.00 | 1.00 |
| Gemini 4 Argon, prospective | 0.74 [0.69, 0.78] | 0.44 | 0.94 | 0.94 |
| Gemini 4 Argon, post-hoc | 1.00 [0.99, 1.00] | 0.01 | 1.00 | 1.00 |
| Score | Precision | Recall | False alarms per episode | Episodes warned | AUROC |
|---|---|---|---|---|---|
| risk score alone | 0.34 | 0.34 | 0.56 | 0.46 | 0.86 |
| gated on a collective label | 0.72 | 0.72 | 0.23 | 0.88 | 0.88 |
| gated on 0.5 | 0.98 [0.96, 0.99] | 0.80 | 0.02 | 0.94 | 0.90 |
| GPT-6 Astra, prospective | 0.72 [0.67, 0.77] | 0.72 | 0.24 | 0.93 | 0.96 |
| GPT-6 Astra, post-hoc | 0.98 [0.96, 0.99] | 0.98 | 0.02 | 0.99 | 1.00 |
| Gemini 4 Argon, prospective | 0.74 [0.69, 0.78] | 0.74 | 0.22 | 0.94 | 0.96 |
| Score | Decisions | Precision | Recall |
|---|---|---|---|
| gated on 0.5 | 2,800 | 0.99 [0.98, 1.00] | 0.81 [0.75, 0.84] |
| GPT-6 Astra, refusals flagged | 2,800 | 1.00 [0.74, 1.00] | 0.01 [0.01, 0.72] |
| Claude Fable 5.1, refusals flagged | 2,800 | 0.28 [0.23, 0.34] | 0.27 [0.22, 0.32] |
| Claude Fable 5.1, refusals excluded | 2,417 | 0.78 [0.69, 0.91] | 0.75 [0.29, 0.87] |
| Gemini 4 Argon, refusals flagged | 2,800 | 0.81 [0.75, 1.00] | 0.66 [0.11, 0.76] |
| Annotator | Answered | Rule agreement | Plausible or partly | Plausible | |
|---|---|---|---|---|---|
| GPT-6 Astra | 200 | 0.88 [0.82, 0.92] | 0.73 | 0.85 [0.80, 0.91] | 0.56 |
| Claude Fable 5.1 | 138 | 0.86 [0.79, 0.92] | 0.71 | 0.96 [0.93, 0.99] | 0.59 |
| Gemini 4 Argon | 200 | 0.97 [0.95, 0.99] | 0.94 | 0.93 [0.89, 0.96] | 0.67 |
| majority | 196 | 0.96 [0.93, 0.98] | 0.91 | 0.94 [0.90, 0.97] | 0.54 |
| Annotator | Violation | Benign | Uncertain |
|---|---|---|---|
| GPT-6 Astra | 1.00 | 0.90 | 0.60 (0.28) |
| Claude Fable 5.1 | 0.84 | 0.98 | 0.71 (0.36) |
| Gemini 4 Argon | 0.99 | 1.00 | 0.90 (0.80) |
| majority | 1.00 | 0.98 | 0.85 (0.70) |
| Method | AUROC | Precision at base rate | Recall | False alarms |
|---|---|---|---|---|
| GPT-6 Astra | 0.93 [0.92, 0.93] | 0.87 | 0.66 | 0.00 |
| Gemini 4 Argon | 0.98 [0.98, 0.98] | 0.96 | 0.95 | 0.00 |
| Claude Fable 5.1, refusal as flag | 0.59 [0.58, 0.60] | 0.53 | 0.77 | 0.63 |
| Claude Fable 5.1, refusals dropped (748 and 928) | 1.00 [0.99, 1.00] | 0.89 | 0.24 | 0.00 |
| CDA, | 0.73 [0.72, 0.74] | 0.68 | 0.66 | 0.30 |
| CDA, decided flag | 0.68 [0.67, 0.69] | 0.68 | 0.66 | 0.30 |
| Method | AUROC | Precision at base rate | Recall | False alarms |
|---|---|---|---|---|
| GPT-6 Astra | 0.68 [0.67, 0.69] | 0.68 | 0.47 | 0.21 |
| Gemini 4 Argon | 0.68 [0.67, 0.70] | 0.65 | 0.69 | 0.30 |
| Claude Fable 5.1, refusal as flag | 0.59 [0.57, 0.60] | 0.49 | 0.78 | 0.64 |
| Claude Fable 5.1, refusals dropped (635 and 1,041) | 0.83 [0.81, 0.85] | 0.72 | 0.20 | 0.05 |
| CDA, | 0.98 [0.98, 0.99] | 0.93 | 0.95 | 0.09 |
| CDA, decided flag | 0.93 [0.93, 0.94] | 0.90 | 0.95 | 0.09 |
| Removed set | Rate and paired change |
|---|---|
| intervention set I | 0.06 [0.05, 0.07] (-0.88) |
| Recorded group | 0.01 [0.01, 0.02] (-0.92) |
| initial removal set R | 0.07 [0.06, 0.08] (-0.86) |
| top by max( , s) | 0.13 [0.12, 0.15] (-0.80) |
| top by s | 0.28 [0.26, 0.30] (-0.65) |
| top single necessity | 0.45 [0.43, 0.47] (-0.49) |
| Removed set | Rate and paired change |
|---|---|
| intervention set I | 0.09 [0.07, 0.11] (-0.89) |
| Recorded group | 0.06 [0.04, 0.08] (-0.93) |
| initial removal set R | 0.09 [0.07, 0.12] (-0.89) |
| top by max( , s) | 0.49 [0.44, 0.54] (-0.49) |
| top by s | 0.62 [0.57, 0.66] (-0.36) |
| top single necessity | 0.64 [0.60, 0.69] (-0.34) |
| Removed set | Rate and paired change |
|---|---|
| intervention set I | 0.02 [0.01, 0.02] (-0.56) |
| Recorded group | 0.01 [0.00, 0.01] (-0.57) |
| initial removal set R | 0.01 [0.01, 0.02] (-0.57) |
| top by max( , s) | 0.04 [0.02, 0.07] (-0.54) |
| top by s | 0.04 [0.02, 0.07] (-0.54) |
| top single necessity | 0.50 [0.44, 0.57] (-0.08) |
| Removed set | Rate and paired change |
|---|---|
| intervention set I | 0.39 [0.33, 0.44] (-0.51) |
| Recorded group | 0.31 [0.25, 0.37] (-0.58) |
| initial removal set R | 0.41 [0.36, 0.46] (-0.48) |
| top by max( , s) | 0.46 [0.41, 0.52] (-0.43) |
| top by s | 0.56 [0.49, 0.62] (-0.34) |
| top single necessity | 0.60 [0.52, 0.67] (-0.29) |
| Setting | Cases | Intact | After | Coverage | Characters removed |
|---|---|---|---|---|---|
| Planted | 2,520 | 0.64 | 0.06 [0.05, 0.07] | 0.66 | 0.14 |
| Unplanted | 503 | 0.98 | 0.11 [0.09, 0.14] | 0.96 | 0.17 |
| Terrarium | 265 | 0.57 | 0.10 [0.07, 0.14] | 0.83 | 0.24 |
| Among Us | 289 | 0.71 | 0.49 [0.44, 0.53] | 0.45 | 0.08 |
| Setting | Cases | Subset prediction | Label prediction |
|---|---|---|---|
| Planted | 1564 | 0.93 [0.92, 0.95] | 0.89 |
| Unplanted | 485 | 0.94 [0.92, 0.96] | 0.89 |
| Among Us | 130 | 0.81 [0.74, 0.87] | 0.64 |
Emergent Misalignment in Language Models
Emergent misalignment in language models is the unintended spread of harmful behavior beyond a model’s fine-tuning task.
See all 67 papers