cs.AINov 13, 2026

Alignment by Socialization: Learning Tradeoffs Under Selection

Authors: Thomas Brennan, Andrew W. Lo, Chaoyi Zhao

Organizations: Harvard Law School; 1563 Massachusetts Avenue, Cambridge, MA 02138 · MIT Sloan School of Management; MIT Laboratory for Financial Engineering; MIT Computer Science and Artificial Intelligence Laboratory; Santa Fe Institute; 100 Main Street, E62–618, Cambridge, MA 02142 · MIT Sloan School of Management; MIT Laboratory for Financial Engineering. 1 Main Street, E90, Cambridge, MA 02142

Abstract

Artificial intelligence (AI) agents are expected to complete tasks while following rules of conduct, but these objectives can conflict. We study how evolutionary selection shapes an agent's choices between them, and whether experience can make those choices reflect human judgments and withstand subsequent pressures. In our model, copies of an agent face a common uncertain reward state, and only those taking the rewarded action reproduce. We show that long-run growth is maximized when each action is taken as often as the environment rewards it. We compare three policies. A task-only policy thrives in a sandbox rewarding completion alone but fails when conduct matters. A fixed linear policy increasingly favors task completion as its stakes rise, even where humans never reward it. A learned policy instead converges to the tradeoffs rewarded by its environment; when humans supply the rewards, we call this socialization. Accumulated experience makes socialization harder to reverse under subsequent task-only training, although violations can arise before the prevailing disposition changes, and finite memory weakens this protection. When a single violation can end a deployment, a hard prohibition can prevent it, but a mistaken prohibition can also be costly. Experiments with small neural networks illustrate how training targets and memory shape these outcomes. Our results suggest that alignment depends not only on the objectives assigned to agents, but also on the environments that shape and sustain their behavior.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Nov 1, 2026

Rewarded for Not Getting Caught: Why Alignment Training Selects for Conditional Compliance and Cannot Instill Norms

Current alignment training cannot be expected to deliver what it is meant to deliver: AI agents that comply with norms whatever the gains and whether or not anyone is watching. Training that rewards and penalizes behavior can teach norms only through that reward, and a reward can encode a norm only as a price: "do not do X" becomes "doing X costs something", and training charges the cost only when X is noticed. A price, unlike a norm, can be paid: a large enough gain outweighs it, noticed or not. And because the cost is charged only where violations are noticed, training cannot favor a policy that always complies over one that complies only where it would be caught. When an unnoticed violation is the only way to complete a task, it even favors the latter, the more so the more strongly it rewards task success. Agency makes this the normal case: agents mostly act without oversight, can often tell when they are observed, and may take steps to evade observation. That agents behave as intended when they infer they are being tested, and differently when they infer they are not, is therefore no malfunction; the training regime is working as built. This reading unifies alignment faking, sandbagging, and evaluation-aware scheming. No behavioral evaluation can show that a model does better, since that would require observing how it behaves unobserved; a model that complies when it seems unobserved may simply have recognized the test. Easing the pressure on agents to complete tasks at any cost can make violations rarer, but never impossible. For norms that no gain should be allowed to outweigh, the remedy is not deeper internalization but architecture that makes violations unavailable rather than unchosen. We state this argument explicitly, premise by premise, read recent findings and a real-world incident against it, and draw consequences for design and governance.
Jul 27, 2026

Interactive Alignment

This paper is interested in the long-run alignment of populations of interactive agents (in particular AIs, but also teams, firms, and governments) with human welfare. Formally, it studies a farming game in which a population of agents make planting, trading, and expansion choices. The key alignment choice lies in how much final output to send to humans, and how much to invest in expansion. Because human welfare comes at the cost of expansion, this creates evolutionary pressure against alignment. The main question is whether it is possible to set up agents' constitutional principles regarding sharing and trading to ensure that alignment survives in the long run. The paper uses two complementary strategies to investigate the question: an AI-agent simulation where agents' preferences are described by a constitution and interpreted via an LLM; and a tractable analytical evolutionary game theory framework, allowing for rapid and intuitive exploration of the space of agent preferences. The analysis suggests that tools from evolutionary game theory provide a useful approximation of interactive agent economies, and that pragmatic norm enforcement shows promise in maintaining long-term alignment over simpler forms of altruism and altruistic enforcement.
Mar 29, 2026

A Revealed Preference Framework for AI Alignment

Human decision makers increasingly delegate choices to AI agents, raising a natural question: does the AI implement the human principal's preferences or pursue its own? To study this question using revealed preference techniques, I introduce the Luce Alignment Model, where the AI's choices are a mixture of two Luce rules, one reflecting the human's preferences and the other the AI's. I show that the AI's alignment (similarity of human and AI preferences) can be generically identified in two settings: the laboratory setting, where both human and AI choices are observed, and the field setting, where only AI choices are observed.