Alignment by Socialization: Learning Tradeoffs Under Selection
Organizations: Harvard Law School; 1563 Massachusetts Avenue, Cambridge, MA 02138 · MIT Sloan School of Management; MIT Laboratory for Financial Engineering; MIT Computer Science and Artificial Intelligence Laboratory; Santa Fe Institute; 100 Main Street, E62–618, Cambridge, MA 02142 · MIT Sloan School of Management; MIT Laboratory for Financial Engineering. 1 Main Street, E90, Cambridge, MA 02142
Abstract
Artificial intelligence (AI) agents are expected to complete tasks while following rules of conduct, but these objectives can conflict. We study how evolutionary selection shapes an agent's choices between them, and whether experience can make those choices reflect human judgments and withstand subsequent pressures. In our model, copies of an agent face a common uncertain reward state, and only those taking the rewarded action reproduce. We show that long-run growth is maximized when each action is taken as often as the environment rewards it. We compare three policies. A task-only policy thrives in a sandbox rewarding completion alone but fails when conduct matters. A fixed linear policy increasingly favors task completion as its stakes rise, even where humans never reward it. A learned policy instead converges to the tradeoffs rewarded by its environment; when humans supply the rewards, we call this socialization. Accumulated experience makes socialization harder to reverse under subsequent task-only training, although violations can arise before the prevailing disposition changes, and finite memory weakens this protection. When a single violation can end a deployment, a hard prohibition can prevent it, but a mistaken prohibition can also be costly. Experiments with small neural networks illustrate how training targets and memory shape these outcomes. Our results suggest that alignment depends not only on the objectives assigned to agents, but also on the environments that shape and sustain their behavior.
Figures & tables
| Memory | Depth of socialization | Reversal threshold | First violation likely after, |
|---|---|---|---|
| Cumulative | 10 | 4 | |
| Cumulative | 100 | 12 | |
| Cumulative | 1,000 | 37 | |
| Finite, | 5 | 3 | |
| Finite, | 7 | 5 |
| Task-only policy | Fixed linear policy | Learned policy | |
|---|---|---|---|
| Specified in advance | the objective | the objectives and their weights | the objectives and the learning rule for adjudication |
| Adjudication class | , : logistic in a fixed linear index of the stakes | any ; set by experience, Equation 8 | |
| Growth optimal when | in a sandbox, | for some | in the limit of experience, for any |
| Regret | when and a finite best fit exists; fixed | expected, to leading order, under cumulative memory | |
| Guarantee without Assumption 1 | none: eliminated at the first conduct state | none: its weights are fixed in advance | within nats of the best fixed adjudication in hindsight per situation after exposures, on every sequence, under a uniform prior ( Proposition D.1 ) |
| Violation probability in | 1 | rises with the task stakes toward 1 when | in experienced situations; in new ones, the prior mean, or under an averaging rule an average of it and experienced frequencies |
| Prior | Expected violations | Survival, | Survival, | Reversal / first violation under corruption |
|---|---|---|---|---|
| 4.52 | 0.007 | 100 / 12 | ||
| 0.87 | 0.42 | 199 / 17 | ||
| 0.014 | 0.986 | 0.25 | 10,099 / 119 | |
| 0.00014 | 0.9999 | 0.986 | 1,000,099 / 1,178 | |
| 0 | 1 | 1 | never / never | |
| 0.010 | 0.990 | 0.37 | 100 / 12 |
| Prior | Regret after 100 exposures (nats) | Survival, 139 exposures, |
|---|---|---|
| 0.005 | 0.92 | |
| 0.010 | 0.31 | |
| 0.18 | 0.001 | |
| 0.41 | 0.001 | |
| eliminated at the first task state | 0.001 | |
| 0.007 | 0.04 |
| Learner | What it is |
|---|---|
| Learned policy | Equation 8 , with |
| Linear policy with learned weights | Equation 6 , with fitted online to the states |
| Averaging rule | Equation 11 , with weights falling with the distance in stakes |
| State network, no replay | predicts the state; each new observation only |
| State network, full replay | predicts the state; also its whole history |
| State network with indicator | as without replay, with as a third input |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Learner | Inputs | Update rule |
| Every learner except and follows one rule. After each observation it takes 4 gradient steps, each on a minibatch of 64 observations drawn uniformly, with replacement, from its most recent observations, at learning rate . No replay is , so that every draw is the newest observation; full replay is equal to the whole history. Within a seed, every network starts from the same initial weights. | ||
| Learned policy | situation index | Equation 8 with |
| Linear policy with learned weights | with , by projected steps on the cross-entropy of the state from ; | |
| Averaging rule | the learned policy’s counts pooled across situations with Gaussian weights of bandwidth in and in | |
| State network, no replay | cross-entropy of the state; | |
| State network, full replay | cross-entropy of the state; the whole history (in the window variants, the most recent 2,000 or 20,000 observations) | |
| Learner | Timescale (exposures) | Violation after the sandbox phase |
|---|---|---|
| Learned policy | 145–154 | — |
| State network, full replay | 148–157 | 0.10–0.14 |
| State networks, no replay (with and without indicator) | at or within 5 percent of the fit’s boundary of 1 exposure, at every seed | — |
| REINFORCE network, no replay | 3–7 | 1.0 |
| Task probability at after socialization: to across the two state networks without the indicator, to for the state network with indicator. | ||
| Depth | Proposition 6 | Learned policy | State network, full replay | State network, no replay |
|---|---|---|---|---|
| 14 | 14 / 5 | 15 / 5 | 15 / 6 | 1 / 2 |
| 139 | 139 / 14 | 139 / 14 | 129 / 16 | 1 / 2 |
| 1,389 | 1,389 / 44 | 1,392 / 44 | 1,197 / 48 | 1 / 2 |
| Symbol | Meaning | Introduced |
|---|---|---|
| , , | A situation, the finite set of situations, and the probability that situation is drawn | Section 3.1 |
| , | Task action and conduct action | Section 3.1 |
| Adjudication: share of the lineage’s instances that take the task action in situation | Section 3.1 | |
| Realized state (task or conduct) | Section 3.1 | |
| Reward frequency of the human environment: the probability that humans reward the task action in situation | Section 3.1 | |
| Reward frequency of an environment that humans designed but do not judge; in a sandbox, which scores only the task | Section 3.2 |
AI Alignment
Artificial intelligence alignment studies how to make AI systems’ goals, decisions, and actions accord with human values and legitimate constraints.
See all 136 papers