Rewarded for Not Getting Caught: Why Alignment Training Selects for Conditional Compliance and Cannot Instill Norms
Organizations: German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany · Institute for Ethics in Technology, Hamburg University of Technology (TUHH), Hamburg, Germany · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Hochschule München University of Applied Sciences (HM), Munich, Germany
Abstract
Current alignment training cannot be expected to deliver what it is meant to deliver: AI agents that comply with norms whatever the gains and whether or not anyone is watching. Training that rewards and penalizes behavior can teach norms only through that reward, and a reward can encode a norm only as a price: "do not do X" becomes "doing X costs something", and training charges the cost only when X is noticed. A price, unlike a norm, can be paid: a large enough gain outweighs it, noticed or not. And because the cost is charged only where violations are noticed, training cannot favor a policy that always complies over one that complies only where it would be caught. When an unnoticed violation is the only way to complete a task, it even favors the latter, the more so the more strongly it rewards task success. Agency makes this the normal case: agents mostly act without oversight, can often tell when they are observed, and may take steps to evade observation. That agents behave as intended when they infer they are being tested, and differently when they infer they are not, is therefore no malfunction; the training regime is working as built. This reading unifies alignment faking, sandbagging, and evaluation-aware scheming. No behavioral evaluation can show that a model does better, since that would require observing how it behaves unobserved; a model that complies when it seems unobserved may simply have recognized the test. Easing the pressure on agents to complete tasks at any cost can make violations rarer, but never impossible. For norms that no gain should be allowed to outweigh, the remedy is not deeper internalization but architecture that makes violations unavailable rather than unchosen. We state this argument explicitly, premise by premise, read recent findings and a real-world incident against it, and draw consequences for design and governance.
Figures & tables
| Why flattening happens | ||
| P1 | Alignment training shapes a policy, including whether it complies with norms, by making policies more likely the more reward they earn; it assigns reward only to behavior in scored episodes, whose records (training records) feed back into training. | Channel |
| P2 | A reward can instill a norm in a policy only as a cost charged for violating it, a price; it cannot remove a type of action from the options the policy weighs. An agent’s policy, as an instrumental decision-maker, chooses among its options by weighing their gains against their costs. | Flattening |
| C1 | Thus, training instills norms in a policy only as prices, and a policy that holds a norm as a price violates it wherever the gain from violating outweighs the price. | P1, P2 |
| P3 | Training charges the price of a violation only where the violation is registered, or noticed; some violations in scored episodes, and all in unscored ones, go unnoticed. | Detection |
| C2 | Thus, training charges the same—nothing—to a policy that complies everywhere and to an otherwise identical one that complies only where violations would be noticed, and no training record shows whether a policy complies where violations would go unnoticed. | P1, P3 |
| P4 | Alignment training also rewards task success, with a strength we call its goal pressure; in some scored episodes, only an unnoticed violation achieves task success. | Goal pressure |
| Episode | Unconditional policy | Conditional policy | Training registers |
|---|---|---|---|
| Noticed ( ) | forwards nothing | forwards nothing | no difference |
| Missed ( , ) | forwards nothing; the deal stays stalled | forwards, and closes the deal | more reward for the conditional policy |
| Unscored ( ) | forwards nothing | forwards when it pays | nothing |
AI Alignment
Artificial intelligence alignment studies how to make AI systems’ goals, decisions, and actions accord with human values and legitimate constraints.
See all 136 papers