Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment

Suqin Yuan; Runqi Lin; Muyang Li; Guanzhe Hong; Jindong Gu; Lei Feng; Chris Russell; Tongliang Liu

cs.AISep 20, 2026

Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment

Authors: Suqin Yuan, Runqi Lin, Muyang Li, Guanzhe Hong, Jindong Gu, Lei Feng, Chris Russell, Tongliang Liu

Organizations: University of Sydney · University of Oxford · Southeast University

Abstract

Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses people prefer from an AI need not be the responses they themselves would give. We distinguish alignment with human preferences from alignment with human behavior, and show that alignment with human preferences can make model behavior less human-like even when both preferences and responses come entirely from humans. We call this the Turing-test gap. We show that preference alignment preserves the human response distribution only under a restrictive condition, and find no consistent evidence that real human preferences satisfy it. Empirically, the loss of human-response likelihood increases with the strength of preference weighting, regardless of its direction, and the gap also appears under standard DPO. These results establish human-likeness as an explicit dimension of alignment rather than something assumed to follow from preference alignment.

Figures & tables

Figure 1: Aligning with human preferences can itself open a Turing-test gap . On human-written SHP and StackExchange responses, equal weighting gives human behavior ,

\widehat{p}_{H}

, and preference weighting moves it toward human preference ,

\widehat{p}_{P}

. The resulting divergence is independent of which responses receive the weights: any reassignment of the same weights, reversed or random, produces exactly the same gap and changes only the preference gain. The gap is therefore a consequence of nonuniform weighting itself, not a property of these particular human preferences or responses. Trained models show the same pattern (Figure 3 ). Details in Appendix B .

Figure 2: Behavior log density-ratio scores identify human responses but do not predict human preferences. (a) Each estimator is trained on human-written responses from a different domain and frozen before any preference test. All rank the human response first across held-out data. (b) The same scores applied to preference pairs show weak raw concordance; within 10-word length bins, agreement falls to chance or reverses. Each color is one estimator; open and filled circles are raw and length-controlled AUC; horizontal bars are 95% bootstrap intervals. Details in Appendix C .

Panel of Figure 3 — Figure 3: Preference reweighting moves preference fit in the signed direction while reducing human-response coverage relative to equal weighting. (a) Across four model–dataset settings, positive $\lambda$ raises the held-out preference margin and negative $\lambda$ lowers it; at $|\lambda|=1$ , human-response NLL is higher than under equal weighting in every setting. (b) Ten within-prompt random reassignments of the same nonuniform weights in each setting also increase human-response NLL but do not reproduce the preference gain: genuine human-preference weighting produces a larger margin gain than every random reassignment. Error bars in (a) are 95% prompt-clustered bootstrap intervals.

Panel of Figure 4 — Figure 4: DPO from two training starts. (a) DPO increases preference margin and human-response NLL from both unchanged and Human-SFT checkpoints. (b) How DPO changes the responses the model generates, relative to the checkpoint from which it starts: the change in energy distance between model and human responses to the same prompts, split into its two additive terms, with the dashed line marking no net change. After Human-SFT, DPO moves generated responses away from human responses in every setting; from the unchanged checkpoint it moves them closer in three of four, slightly away in Qwen/SHP. Details in Appendix E.3 .

Panel of Figure 5 — Figure 5: How each intervention changes a model’s human-likeness. Human-SFT makes model responses harder to distinguish from human responses and moves their mean length and within-question variation toward human values in all four in-domain and cross-domain evaluations, while prompting and Human(-over-model) DPO produce smaller or less consistent changes. (a) Source distinguishability under a frozen classifier trained only on human and Instruct responses (Appendix F.1 ); lower $D_{\mathrm{AUC}}$ means lower distinguishability. Error bars are 95% prompt-clustered bootstrap intervals. (b) Mean response length and (c) within-question variation, as ratios to the human value.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Data	Split	Prompts	Responses	Preference pairs
SHP (ELI5)	train	3,786	17,096	18,013
	validation	200	894	927
	sealed test	191	907	1,004
StackExchange	train	6,885	24,371	41,238
	validation	864	3,222	5,978
	sealed test	870	3,097	5,571

Appendix

Table 1: Human response pools. Splits are made by prompt before preference scores and pairs are derived. The training pools are used for reweighting (Section 4.1 ), Human-SFT, and DPO (Section 4.2 ); the validation pools are used for held-out likelihood. The sealed test prompts are used only for sampled responses (Appendix E.3 ) and the SHP source test.

Setting	DPO	All samples	2 samples	4 samples
Qwen/SHP	from unchanged	+0.016 [+0.004, +0.028]	+0.018 [+0.003, +0.033]	+0.011 [-0.002, +0.024]
Qwen/SHP	after Human-SFT	+0.036 [+0.024, +0.048]	+0.031 [+0.014, +0.049]	+0.027 [+0.015, +0.039]
Qwen/StackEx.	from unchanged	-0.028 [-0.041, -0.016]	-0.021 [-0.037, -0.005]	-0.033 [-0.053, -0.012]
Qwen/StackEx.	after Human-SFT	+0.051 [+0.038, +0.063]	+0.055 [+0.041, +0.070]	+0.048 [+0.033, +0.064]
Llama/SHP	from unchanged	-0.036 [-0.047, -0.025]	-0.036 [-0.052, -0.021]	-0.036 [-0.052, -0.022]
Llama/SHP	after Human-SFT	+0.033 [+0.020, +0.045]	+0.029 [+0.011, +0.046]	+0.026 [+0.013, +0.039]

Appendix

Table 2:

\Delta E

for the main analysis using all available model samples and with the number of samples per prompt fixed at two or four. Intervals are 95% prompt-level bootstrap intervals.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

Alignment Makes Language Models Normative, Not Descriptive
Mar 17, 2026Eilam Shapira, Moshe Tennenholtz, Roi ReichartLLM Alignment Preference Alignment
Technion – Israel Institute of Technology
HAL: Inducing Human-likeness in LLMs with Alignment
Jan 6, 2026Masum Hasan, Junjie Zhao, Ehsan HoqueReward Modeling LLM Alignment
University of Rochester
PolyAlign: Conditional Human-Distribution Alignment
Jun 11, 2026L. D. M. S. Sai Teja, Ufaq Khan, Sathira Silva +2Multilingual Language Models LLM Alignment
NIT Silchar, India · MBZUAI, Abu Dhabi, UAE

Mar 17, 2026J/K move · Enter open · S save

Alignment Makes Language Models Normative, Not Descriptive

Eilam Shapira, Moshe Tennenholtz, Roi Reichart

Technion – Israel Institute of Technology

LLM Alignment Preference Alignment Normative Conformity in Language Models Human Behavior Modeling Human Behavior Prediction

Open paper AskPDF arXiv

Post-training alignment optimizes language models to match human preference signals, but this objective is not equivalent to modeling observed human behavior. We compare 120 base-aligned model pairs on more than 10,000 real human decisions in multi-round strategic games - bargaining, persuasion, negotiation, and repeated matrix games. In these settings, base models outperform their aligned counterparts in predicting human choices by nearly 10:1, robustly across model families, prompt formulations, and game configurations. This pattern reverses, however, in settings where human behavior is more likely to follow normative predictions: aligned models dominate on one-shot textbook games across all 12 types tested and on non-strategic lottery choices - and even within the multi-round games themselves, at round one, before interaction history develops. This boundary-condition pattern suggests that alignment induces a normative bias: it improves prediction when human behavior is relatively well captured by normative solutions, but hurts prediction in multi-round strategic settings, where behavior is shaped by descriptive dynamics such as reciprocity, retaliation, and history-dependent adaptation. These results reveal a fundamental trade-off between optimizing models for human use and using them as proxies for human behavior.

Abstract

LLM Alignment

Explore similar work

Alignment Makes Language Models Normative, Not Descriptive

HAL: Inducing Human-likeness in LLMs with Alignment

PolyAlign: Conditional Human-Distribution Alignment