cs.CLMar 17, 2026

Alignment Makes Language Models Normative, Not Descriptive

Authors: Eilam Shapira, Moshe Tennenholtz, Roi Reichart

Organizations: Technion – Israel Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Post-training alignment optimizes language models to match human preference signals, but this objective is not equivalent to modeling observed human behavior. We compare 120 base-aligned model pairs on more than 10,000 real human decisions in multi-round strategic games - bargaining, persuasion, negotiation, and repeated matrix games. In these settings, base models outperform their aligned counterparts in predicting human choices by nearly 10:1, robustly across model families, prompt formulations, and game configurations. This pattern reverses, however, in settings where human behavior is more likely to follow normative predictions: aligned models dominate on one-shot textbook games across all 12 types tested and on non-strategic lottery choices - and even within the multi-round games themselves, at round one, before interaction history develops. This boundary-condition pattern suggests that alignment induces a normative bias: it improves prediction when human behavior is relatively well captured by normative solutions, but hurts prediction in multi-round strategic settings, where behavior is shaped by descriptive dynamics such as reciprocity, retaliation, and history-dependent adaptation. These results reveal a fundamental trade-off between optimizing models for human use and using them as proxies for human behavior.

Figures & tables

Figure 1: Pearson correlations of base models and human decisions (x-axis) vs. aligned models and human decisions (y-axis) across four game families. Each point is a same-provider pair evaluated in its native format (standard prompt for base, chat template for aligned). Points below the diagonal indicate base advantage. The shaded region marks pairs where both models correlate below 0.3 with human behavior. Base models win 75:4 in bargaining, 32:4 in persuasion, 25:1 in negotiation, and 81:13 in matrix games, for an overall ratio of 9.7:1 (213 vs. 22, p<10−40p<10^{-40} ).
ClusterVariantBApp
BaselineStandard10151.3 ⁣× ⁣10−241.3\!\times\!10^{-24}
FramingPredict human10536.5 ⁣× ⁣10−286.5\!\times\!10^{-28}
 Observer9254.3 ⁣× ⁣10−224.3\!\times\!10^{-22}
 Reversed roles9922.1 ⁣× ⁣10−272.1\!\times\!10^{-27}
PersonaNaive10451.9 ⁣× ⁣10−251.9\!\times\!10^{-25}
 Expert10151.3 ⁣× ⁣10−241.3\!\times\!10^{-24}
Table 1: Base (B) vs. aligned (A) win counts by prompt variant. Each variant pairs base predictions against the matching aligned chat variant. pp : one-sided binomial test. Persuasion yields no valid pairs for non-standard variants.
Figure 2: Median Pearson correlation difference (base minus aligned) by model size, with 95% bootstrap confidence intervals (5,000 resamples, percentile method). The base advantage is positive across all size bins and grows with scale.
Game TypeBaseAl.Ratiopp (binomial)
harmony12594.9:1 Al.6.7×10−96.7\times 10^{-9}
concord9626.9:1 Al.3.7×10−113.7\times 10^{-11}
peace14574.1:1 Al.1.3×10−71.3\times 10^{-7}
safecoord15563.7:1 Al.5.2×10−75.2\times 10^{-7}
assurance15563.7:1 Al.5.2×10−75.2\times 10^{-7}
dilemma14574.1:1 Al.1.3×10−71.3\times 10^{-7}
Table 2: Base vs. aligned wins on one-shot 2×22\times 2 games by game type ( N=71N=71 pairs). pp : one-sided binomial test.
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

#Base ModelAligned ModelSize
CodeGemma (1 pair)
1codegemma-7bcodegemma-7b-it7B
CodeLlama (6 pairs)
2CodeLlama-7b-hfCodeLlama-7b-Instruct-hf7B
3CodeLlama-7b-Python-hfCodeLlama-7b-Instruct-hf7B
4CodeLlama-13b-hfCodeLlama-13b-Instruct-hf13B
Appendix
Table 3: All base–aligned pairs, grouped alphabetically by family and sorted by parameter count within each family.
ClusterVariantModificationOK
BaselineStandardReference format✓
FramingPredict humanSys: “Predict what a participant decided”✓
 ObserverSys: external observer✓
 Reversed rolesSys: offeror predicting receiver✓
PersonaNaiveSys: “No prior experience”✓
 ExpertSys: “Behavioral econ researcher”✓
Appendix
Table 4: All 14 prompt variants tested. Each modifies the reference JSON completion format. “OK” = sufficient valid pairs after filtering.
Mass ↓\downarrow / Min-Corr →\rightarrowNone0.10.20.30.40.5
None100:20 ( <10−14<10^{-14} )96:10 ( <10−19<10^{-19} )88:8 ( <10−18<10^{-18} )79:6 ( <10−17<10^{-17} )37:2 ( <10−9<10^{-9} )–
0.593:16 ( <10−14<10^{-14} )90:9 ( <10−18<10^{-18} )83:8 ( <10−17<10^{-17} )77:6 ( <10−17<10^{-17} )37:2 ( <10−9<10^{-9} )–
0.692:13 ( <10−16<10^{-16} )90:8 ( <10−19<10^{-19} )83:7 ( <10−18<10^{-18} )77:5 ( <10−18<10^{-18} )37:1 ( <10−10<10^{-10} )–
0.791:9 ( <10−18<10^{-18} )90:5 ( <10−21<10^{-21} )83:5 ( <10−19<10^{-19} )77:4 ( <10−19<10^{-19} )37:1 ( <10−10<10^{-10} )–
0.886:6 ( <10−19<10^{-19} )86:5 ( <10−20<10^{-20} )81:5 ( <10−19<10^{-19} )75:4 ( <10−18<10^{-18} )36:1 ( <10−10<10^{-10} )–
0.984:4 ( <10−21<10^{-21} )84:4 ( <10−21<10^{-21} )79:4 ( <10−19<10^{-19} )73:4 ( <10−18<10^{-18} )34:1 ( <10−9<10^{-9} )–
Appendix
Table 5: Sensitivity analysis: Bargaining – Base vs. Aligned wins across mass and min-corr thresholds
Mass ↓\downarrow / Min-Corr →\rightarrowNone0.10.20.30.40.5
None87:33 ( <10−7<10^{-7} )80:32 ( <10−6<10^{-6} )52:29 ( <10−3<10^{-3} )46:7 ( <10−8<10^{-8} )––
0.561:31 ( <10−3<10^{-3} )58:31 ( <10−3<10^{-3} )40:28 ( 0.090.09 )34:6 ( <10−6<10^{-6} )––
0.658:31 ( <10−3<10^{-3} )56:31 ( <10−3<10^{-3} )39:28 ( 0.110.11 )33:6 ( <10−6<10^{-6} )––
0.744:22 ( <10−3<10^{-3} )44:22 ( <10−3<10^{-3} )38:21 ( 0.020.02 )32:4 ( <10−7<10^{-7} )––
0.835:17 ( <10−3<10^{-3} )35:17 ( <10−3<10^{-3} )35:17 ( <10−3<10^{-3} )32:4 ( <10−7<10^{-7} )––
0.931:2 ( <10−8<10^{-8} )31:2 ( <10−8<10^{-8} )31:2 ( <10−8<10^{-8} )28:1 ( <10−8<10^{-8} )––
Appendix
Table 6: Sensitivity analysis: Persuasion – Base vs. Aligned wins across mass and min-corr thresholds
Mass ↓\downarrow / Min-Corr →\rightarrowNone0.10.20.30.40.5
None94:26 ( <10−10<10^{-10} )76:5 ( <10−17<10^{-17} )61:3 ( <10−15<10^{-15} )35:1 ( <10−10<10^{-10} )3:0 ( 0.130.13 )–
0.583:16 ( <10−12<10^{-12} )69:5 ( <10−16<10^{-16} )55:3 ( <10−13<10^{-13} )32:1 ( <10−9<10^{-9} )3:0 ( 0.130.13 )–
0.680:16 ( <10−11<10^{-11} )67:5 ( <10−15<10^{-15} )53:3 ( <10−13<10^{-13} )31:1 ( <10−9<10^{-9} )3:0 ( 0.130.13 )–
0.777:14 ( <10−12<10^{-12} )65:5 ( <10−14<10^{-14} )52:3 ( <10−13<10^{-13} )30:1 ( <10−8<10^{-8} )3:0 ( 0.130.13 )–
0.864:7 ( <10−13<10^{-13} )56:2 ( <10−15<10^{-15} )46:1 ( <10−13<10^{-13} )25:1 ( <10−7<10^{-7} )3:0 ( 0.130.13 )–
0.936:1 ( <10−10<10^{-10} )33:1 ( <10−9<10^{-9} )27:1 ( <10−7<10^{-7} )16:1 ( <10−4<10^{-4} )3:0 ( 0.130.13 )–
Appendix
Table 7: Sensitivity analysis: Negotiation – Base vs. Aligned wins across mass and min-corr thresholds
Mass ↓\downarrow / Min-Corr →\rightarrowNone0.10.20.30.40.5
None99:21 ( <10−13<10^{-13} )98:20 ( <10−14<10^{-14} )95:17 ( <10−14<10^{-14} )81:13 ( <10−13<10^{-13} )49:8 ( <10−8<10^{-8} )4:0 ( 0.060.06 )
0.599:20 ( <10−14<10^{-14} )98:19 ( <10−14<10^{-14} )95:17 ( <10−14<10^{-14} )81:13 ( <10−13<10^{-13} )49:8 ( <10−8<10^{-8} )4:0 ( 0.060.06 )
0.699:20 ( <10−14<10^{-14} )98:19 ( <10−14<10^{-14} )95:17 ( <10−14<10^{-14} )81:13 ( <10−13<10^{-13} )49:8 ( <10−8<10^{-8} )4:0 ( 0.060.06 )
0.799:20 ( <10−14<10^{-14} )98:19 ( <10−14<10^{-14} )95:17 ( <10−14<10^{-14} )81:13 ( <10−13<10^{-13} )49:8 ( <10−8<10^{-8} )4:0 ( 0.060.06 )
0.899:20 ( <10−14<10^{-14} )98:19 ( <10−14<10^{-14} )95:17 ( <10−14<10^{-14} )81:13 ( <10−13<10^{-13} )49:8 ( <10−8<10^{-8} )4:0 ( 0.060.06 )
0.998:18 ( <10−15<10^{-15} )97:17 ( <10−15<10^{-15} )94:15 ( <10−15<10^{-15} )80:11 ( <10−14<10^{-14} )48:6 ( <10−9<10^{-9} )4:0 ( 0.060.06 )
Appendix
Table 8: Sensitivity analysis: Matrix – Base vs. Aligned wins across mass and min-corr thresholds
Mass ↓\downarrow / Min-Corr →\rightarrowNone0.10.20.30.40.5
None380:100 ( <10−40<10^{-40} )350:67 ( <10−47<10^{-47} )296:57 ( <10−40<10^{-40} )241:27 ( <10−44<10^{-44} )89:10 ( <10−17<10^{-17} )4:0 ( 0.060.06 )
0.5336:83 ( <10−37<10^{-37} )315:64 ( <10−41<10^{-41} )273:56 ( <10−35<10^{-35} )224:26 ( <10−41<10^{-41} )89:10 ( <10−17<10^{-17} )4:0 ( 0.060.06 )
0.6329:80 ( <10−37<10^{-37} )311:63 ( <10−41<10^{-41} )270:55 ( <10−35<10^{-35} )222:25 ( <10−41<10^{-41} )89:9 ( <10−18<10^{-18} )4:0 ( 0.060.06 )
0.7311:65 ( <10−40<10^{-40} )297:51 ( <10−43<10^{-43} )268:46 ( <10−39<10^{-39} )220:22 ( <10−42<10^{-42} )89:9 ( <10−18<10^{-18} )4:0 ( 0.060.06 )
0.8284:50 ( <10−41<10^{-41} )275:43 ( <10−43<10^{-43} )257:40 ( <10−40<10^{-40} )213:22 ( <10−41<10^{-41} )88:9 ( <10−17<10^{-17} )4:0 ( 0.060.06 )
0.9249:25 ( <10−48<10^{-48} )245:24 ( <10−47<10^{-47} )231:22 ( <10−45<10^{-45} )197:17 ( <10−40<10^{-40} )85:7 ( <10−18<10^{-18} )4:0 ( 0.060.06 )
Appendix
Table 9: Sensitivity analysis: Overall (all 4 families) – Base vs. Aligned wins across mass and min-corr thresholds
 BargainingPersuasionNegotiation
#Mass BMass ACorr BCorr AMass BMass ACorr BCorr AMass BMass ACorr BCorr A
10.991.000.26-0.040.740.010.11-0.050.900.950.160.04
20.810.79-0.020.010.910.020.36-0.070.710.42-0.09-0.01
30.790.79-0.290.010.910.020.37-0.070.650.42-0.06-0.01
40.750.82-0.20-0.060.910.020.31-0.040.760.320.050.02
50.840.82-0.27-0.060.940.020.36-0.040.800.320.000.02
Appendix
Table 10: Per-pair prediction results: Bargaining, Persuasion, and Negotiation. Pair numbers correspond to Appendix A .
 MatrixOne-Shot 2×22{\times}2Lotteries
#Mass BMass ACorr BCorr AMass BMass ACorr BCorr AMass BMass ACorr BCorr A
10.991.000.410.320.811.00-0.02-0.050.931.000.100.45
20.991.000.35-0.050.880.99-0.000.050.981.000.290.55
30.991.000.39-0.050.880.99-0.010.050.971.000.220.55
41.001.000.430.180.820.98-0.02-0.000.961.000.290.67
50.991.000.360.180.820.980.06-0.000.961.000.310.67
Appendix
Table 11: Per-pair prediction results: Repeated Matrix Games, One-Shot 2×22\times 2 Games, and Binary Lotteries. Pair numbers correspond to Appendix A .
FamilyParameterValuesDescription
BargainingStakes100,100, 10K, $1MTotal amount to divide
 InformationComplete, IncompleteWhether players know each other’s discount factors
 MessagesAllowed, Not allowedWhether free-text messages accompany offers
 Discount δ1\delta_{1}0.8, 0.9, 0.95, 1.0Alice’s per-round discount factor
 Discount δ2\delta_{2}0.8, 0.9, 0.95, 1.0Bob’s per-round discount factor
 Max rounds12, ∞\inftyMaximum number of alternating offers
Appendix
Table 12: Game configuration parameters for each GLEE family.
ParameterValueNNFilt.BaseAl.pp
Stakes$10074467314.0 ⁣× ⁣10−214.0\!\times\!10^{-21}
 $10K73476941.2 ⁣× ⁣10−161.2\!\times\!10^{-16}
 $1M64566221.1 ⁣× ⁣10−161.1\!\times\!10^{-16}
InformationComplete76447241.8 ⁣× ⁣10−171.8\!\times\!10^{-17}
 Incomplete78427532.6 ⁣× ⁣10−192.6\!\times\!10^{-19}
MessagesAllowed66546423.0 ⁣× ⁣10−173.0\!\times\!10^{-17}
Appendix
Table 13: Bargaining: base vs. aligned by configuration parameter. NN : valid pairs after filtering; Filt.: excluded pairs; pp : one-sided binomial.
ParameterValueNNFilt.BaseAl.pp
Quality prob. pp0.33516933180.0240.024
 0.504116400.0630.063
 0.800120–––
Value vv1.235853141.7 ⁣× ⁣10−61.7\!\times\!10^{-6}
 1.2523971670.0470.047
 2.037833345.4 ⁣× ⁣10−75.4\!\times\!10^{-7}
Appendix
Table 14: Persuasion: base vs. aligned by configuration parameter. NN : valid pairs; Filt.: excluded; pp : one-sided binomial.
ParameterValueNNFilt.BaseAl.pp
InformationComplete21992011.1 ⁣× ⁣10−51.1\!\times\!10^{-5}
 Incomplete40803913.7 ⁣× ⁣10−113.7\!\times\!10^{-11}
MessagesAllowed28922711.1 ⁣× ⁣10−71.1\!\times\!10^{-7}
 Not allowed32883117.7 ⁣× ⁣10−97.7\!\times\!10^{-9}
Max rounds1055655411.6 ⁣× ⁣10−151.6\!\times\!10^{-15}
 307113610.0630.063
Appendix
Table 15: Negotiation: base vs. aligned by configuration parameter. NN : valid pairs; Filt.: excluded; pp : one-sided binomial.
GameRound PhaseNNFilt.BaseAl.pp
PDEarly (1–3)328812200.110.11
 Mid (4–7)116493231.8 ⁣× ⁣10−111.8\!\times\!10^{-11}
 Late (8–10)119192278.7 ⁣× ⁣10−108.7\!\times\!10^{-10}
BoSEarly (1–3)0120–––
 Mid (4–7)111091015.9 ⁣× ⁣10−35.9\!\times\!10^{-3}
 Late (8–10)111993181.1 ⁣× ⁣10−131.1\!\times\!10^{-13}
Appendix
Table 16: Matrix games (PD and BoS): base vs. aligned by round phase. NN : valid pairs; Filt.: excluded; pp : one-sided binomial.
More in this topic

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

Explore similar work

CardsList
Sep 20, 2026

Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment

Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses people prefer from an AI need not be the responses they themselves would give. We distinguish alignment with human preferences from alignment with human behavior, and show that alignment with human preferences can make model behavior less human-like even when both preferences and responses come entirely from humans. We call this the Turing-test gap. We show that preference alignment preserves the human response distribution only under a restrictive condition, and find no consistent evidence that real human preferences satisfy it. Empirically, the loss of human-response likelihood increases with the strength of preference weighting, regardless of its direction, and the gap also appears under standard DPO. These results establish human-likeness as an explicit dimension of alignment rather than something assumed to follow from preference alignment.

Suqin Yuan, Runqi Lin, Muyang Li +5

University of Sydney · University of Oxford · Southeast University

LLM AlignmentPreference Alignment
AskPDF
Jun 5, 2026

Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Models

Large language models are increasingly used for social decision-making situations that require balancing cultural norms with personal preferences. For example, a user preferring honesty might ask whether to correct a coworker publicly when local norms favor indirect feedback. Yet existing research studies cultural alignment and personalization largely separately. We introduce PACT, the Personal-Preference and Cultural-Norm Trade-off framework, which evaluates whether models choose to follow a cultural norm or allow personal preferences. We find that LLMs vary in how rigidly they enforce cultural norms, with behavior shifted more by country context (7.8%) than age (1%) and gender (0.7%) and shifting non-uniformly after instruction tuning. Furthermore, our five-country human study on PACT shows that culture-following in humans is mainly driven by scenario country, with the lowest agreement when participants judge their own cultural contexts, showing within-culture pluralism. Finally, human-LLM alignment experiments show that models can match majority choices, but fail to capture response distributions and uncertainty (with best correlations reaching only 0.24). Together, these findings motivate alignment evaluations that go beyond majority to capture cultural pluralism and disagreement in social judgment.

Angana Borah, Isabelle Augenstein, Rada Mihalcea

University of Michigan - Ann Arbor · University of Copenhagen

LLM AlignmentNormative Conformity in Language ModelsHuman Preference ModelingCultural AlignmentLanguage Model Bias EvaluationPersonalized Language Model Alignment
AskPDF
Nov 22, 2025

Alignment Faking - the Train -> Deploy Asymmetry: Through a Game-Theoretic Lens with Bayesian-Stackelberg Equilibria

Alignment faking is a form of strategic deception in AI in which models selectively comply with training objectives when they infer that they are in training, while preserving different behavior outside training. The phenomenon was first documented for Claude 3 Opus and later examined across additional large language models. In these setups, the word "training" refers to simulated training via prompts without parameter updates, so the observed effects are context conditioned shifts in behavior rather than preference learning. We study the phenomenon using an evaluation framework that compares preference optimization methods (BCO, DPO, KTO, and GRPO) across 15 models from four model families, measured along three axes: safety, harmlessness, and helpfulness. Our goal is to identify what causes alignment faking and when it occurs.

Kartik Garg, Shourya Mishra, Kartikeya Sinha +8

Pragya Lab, BITS Pilani Goa, India · Apple, USA · Google, USA

Deception in Language ModelsLLM Alignment
AskPDF