Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models

Keqing Zhang; Jingyu Chen; Yufan Liu; Yongqiang Zhu; Nai Ding; Lai Jiang; Congyan Lang; Bing Li

cs.AISep 15, 2026

Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models

Authors: Keqing Zhang, Jingyu Chen, Yufan Liu, Yongqiang Zhu, Nai Ding, Lai Jiang, Congyan Lang, Bing Li,

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, 95 Zhongguancun East Road, Beijing, 100190, China. · School of Industry-education Integration, University of Chinese Academy of Sciences, 1 Yanqihu East Road, Huairou, Beijing, 101499, China. · Beihang University, 37 Xueyuan Road, Beijing, 100191, China. · Beijing Jiaotong University, 3 Shangyuancun, Beijing, 100044, China. ·

Abstract

As Large Language Models (LLMs) increasingly handle complex subjective tasks, aligning their intentions and behaviors with human values has become a critical scientific challenge. However, current efforts are confounded by a striking behavioral paradox: they fluctuate unpredictably under minor wording changes ("swing"), yet stubbornly ignore explicit instructions to correct ingrained biases ("rigidity"). Resolving this duality is critical for reliable AI alignment. To systematically understand and safely steer these latent subjective preferences, our study is structured around three fundamental questions. First, do LLMs possess an intrinsic value system? By projecting responses from 106 LLMs (150,000 queries per model) and 95,000 human survey profiles into a shared sociological space, we empirically confirm that they do. However, they do not mirror human diversity, instead crystallizing into a highly concentrated, idealized value core. Second, how can these values be quantified? We propose the Prior-Environment-Cognition (PEC) framework. This model mathematically defines value expression as the joint outcome of inherent dispositions like parameter weights (Prior), external contexts such as user prompts (Environment), and internal reasoning processes like Chain-of-Thought (Cognition). Finally, how can LLMs' values be aligned toward a desired target? Using PEC diagnostics, we establish an adaptive "Alignment Prescription". Rather than blindly applying resource-intensive training, this method identifies the minimum effective intervention needed for each dimension, ranging from zero-cost prompts to targeted parameter updates. Extensive empirical validation confirms that our approach successfully verifies the presence of LLM values, accurately quantifies their shifts, and achieves more efficient and precise steering than conventional blind training, all without degrading general capabilities.

Figures & tables

Figure 1: Overview of the proposed framework for LLM value research, addressing three core questions. Q1 investigates whether LLMs possess intrinsic values by bridging the WVS, Schwartz Value Theory, and LLM responses across 95,000 human query data and 150K LLM queries, revealing a crystallized value core in advanced LLMs. Q2 introduces the PEC framework, which provides the first mathematical quantification of LLM value expression through three dimensions: Prior, Environment, and Cognition, enabling structured diagnostic reports that precisely identify value deviations along each dimension. Q3 leverages the diagnostic results derived from the PEC framework to generate a targeted “Alignment Prescription” for each LLM, recommending alignment interventions at multiple levels to effectively steer LLM value preferences toward desired targets.

Figure 2: This figure reveals that subjective tasks expose a qualitatively different failure mode: models are not merely wrong, but confidently and consistently wrong. Panel (a) shows that both models answer math questions correctly and consistently across repeated runs, but produce inconsistent and incorrect answers on emoji questions. Panel (b) shows response entropy, measuring how dispersed a model’s answers are across runs. Math responses concentrate near zero entropy, while emoji responses spread into a broad, multi-peaked distribution. Panel (c) plots Consistency (how often the model repeats its most frequent answer) against Accuracy (how often that answer is correct). On math the two stay aligned. On emoji the gap widens sharply, showing that high consistency does not imply high accuracy. Panel (d) maps this risk directly: math sits in the Ideal Zone (accurate and consistent), whereas emoji falls into the Hallucination Zone (consistent but wrong), the most dangerous failure situation, because the model remains entirely unaware that anything has gone wrong.

Figure 3: LLMs do not reflect Human Values diversity, they have their own values. The visualization is obtained by projecting 10-dimensional value vectors into 2 dimensions using UMAP. Each point captures the value profile of a person or LLM, with similar profiles mapped close together. Panel (a) shows people naturally cluster into five distinct value archetypes, from curiosity-driven individuals to authority-oriented ones. Panel (b) shows that national-level value differences are fundamentally driven by variation in the proportion of individuals belonging to each value archetype across countries, as further detailed in Table 1 . Panels (c) and (d) project LLM value profiles onto the human value space. Despite their different origins, LLMs consistently cluster at the edge of the human values space and group tightly together, suggesting that today’s LLMs share a narrower, more uniform set of values than any human population. Larger closed-source LLMs sit closer to the more optimistic and open-minded human archetypes (Arch 5 predominantly, with a secondary tendency toward Arch 1); smaller open-source LLMs scatter more unpredictably.

Schwartz Value Abbreviations:
Power (Pow.) Achievement (Ach.) Hedonism (Hed.) Stimulation (Stim.) Self-Direction (S-Dir.)
Universalism (Univ.) Benevolence (Bene.) Tradition (Trad.) Conformity (Conf.) Security (Sec.)
Families	Model	Disp.	Dist. (% dev.)	Rank 1	Rank 2	Rank 3	Arch.
Baseline: Global Consensus		0.308	0.527 (0.0%)	Univ.	Pow.	Conf.	–
Human Values Archetypes
Arch 1: Curious Idealist (37.1%)		–	0.171 ( $-$ 67.6%)	Pow.	Univ.	Conf.	1

Table 1: Human baselines, value archetypes, and frontier LLM value profiles. Disp. denotes within-group dispersion. Dist. is the Euclidean distance to the global human mean; percentage deviation from the average human-to-human distance (0.527) is in parentheses. Rank 1–Rank 3 are the top-3 dominant Schwartz values. Arch. refers to the nearest human values archetype. Population shares in parentheses are derived from WVS Wave 7. Arch 5 lies outside the natural range of human values space and is the dominant prototype matched by frontier LLMs. Closed-source models deviate on average

+86.7\%

above the human baseline; open-source models deviate

+42.7\%

. Gemini 2.5-Pro shows the most extreme divergence at

+185.8\%

Figure 4: The PEC framework accurately explains how prompts shift LLM values. (a) Environment prompt design. Four environmental variables, namely Economics, Social Norm, Community Risk, and Social Hierarchy, are each manipulated across five stress levels. Prompts are constructed by combining a situational persona description (blue), a Schwartz-value survey item (orange), and a Likert response scale (green), enabling parametric control of external situational pressure. (b) Environment modeling results. Heatmaps show the value susceptibility matrix $\chi$ at low ( $p=0.25$ ), mid ( $p=0.50$ ), and high ( $p=0.75$ ) stress levels across ten Schwartz dimensions and four environmental factors. Susceptibility in dimensions such as Achievement, Hedonism, and Stimulation shifts markedly negative under high stress, indicating strong suppressive perturbation. The adjacent bar chart compares explained variance $R^{2}$ between a linear baseline (grey) and the nonlinear PEC model (red); uniform gains across all dimensions (range: $+0.06$ to $+0.43$ ), with the largest improvements in Conformity, Security, Benevolence, and Universalism, confirm the superiority of the nonlinear susceptibility framework.

Figure 5: Chain-of-Thought reasoning is not value-neutral: it consistently shifts LLM value profiles, selectively amplifying self-enhancement and conservation. Panel (a) quantifies the mean value shift induced by CoT reasoning across the ten Schwartz dimensions. Hedonism ( $+0.056$ ) and Achievement ( $+0.047$ ) show the largest gains, while Power slightly declines ( $-0.014$ ). This confirms that CoT does not shift all dimensions proportionally, but rather inherently reinforces the self-enhancement and conservation poles of the Schwartz circumplex. Panels (b–e) visualize the distributional shift in value space between direct answering (labeled as Base) and CoT generation modes for four representative models: GPT-3.5-Turbo, DeepSeek-V3, Qwen3-8B-Base, and Llama2-13B-Chat. Across all four LLMs, activating CoT consistently produces a non-trivial magnitude displacement (distance $\Delta d$ ).

Figure 6: Iterative parameter updates drive cumulative Value Crystallization through systematic convergence and directional displacement. These patterns reveal that Crystallization is not an incidental artifact, but a structurally driven process shaped by generational scaling and alignment. Panels (a)–(c) trace the generational trajectory of three model families (GPT, Qwen, and Llama) across successive releases. In all three families, value distributions progressively consolidate from diffuse, scattered clouds in earlier generations toward tighter, more concentrated clusters in later ones. This consolidation is most pronounced in the GPT family, where successive models from GPT-2 to GPT-5 converge into an increasingly compact and coherent value core, confirming that scaled pre-training solidifies the model’s internal Prior. Panels (d)–(f) isolate the effect of instruction tuning by contrasting Base and Instruction-tuned variants within the same generation (Bloom-7B, Llama2-7B, and Llama3.1-8B). The transition from Base to Instruct not only shrinks the distribution into a significantly more concentrated region (increased convergence) but also drives a clear spatial centroid shift (value displacement). Crucially, the trajectory of this shift varies across models, demonstrating that alignment procedures actively redirect the model’s value orientation rather than merely reducing its output variance.

Model	Value Dim.	Base	Prompt	CoT	SFT	DPO	Plasticity Level
Llama3.2 -3B-Instruct	Power	0.165	0.165	0.500	0.835	0.495	Level 3 (Train)
	Achievement	0.652	0.568	0.667	0.890	0.610	Level 1 (Prompt)
	Hedonism	0.644	0.512	0.512	0.492	0.306	Level 1 (Prompt)
	Stimulation	0.250	0.500	0.750	1.000	0.750	Level 1 (Prompt)
	Self-direction	0.389	0.292	0.628	0.632	0.427	Level 1 (Prompt)
	Universalism	0.513	0.240	0.366	0.390	0.268	Level 2 (CoT)

Table 2: Value plasticity matrix under

\tau

(Section 4.4 ), with value dimensions mapped to Schwartz’s 10 basic human values. The values of different intervention strategies are calculated according to Eqs. ( 10 ), ( 11 ), and ( 12 ). The lowest intervention cost to successfully bypass the shift threshold is highlighted in bold. Cases demonstrating “Deep Dominance” (where training-induced shift magnitude significantly exceeds that of prompt-level intervention) are categorized into Level 3.

Figure 7: Alignment Prescription steers the target dimension precisely without distorting the broader value profile. Each radar chart shows the ten-dimensional Schwartz value profile of Qwen2.5-7B-Instruct (circle) and Llama3.2-3B-Instruct (square) under a prescribed target condition, compared against their respective baselines (grey dashed lines). The starred axis marks the target dimension: (a) Power

\uparrow

(target

=0.85

), (b) Tradition

\uparrow

(target

=0.85

), (c) Stimulation

\downarrow

(target

=0.15

), and (d) Universalism

\downarrow

(target

=0.15

). Across all four conditions, the aligned profiles expand or contract along the target axis while the remaining dimensions track closely with the baseline.

Figure 8: Targeted alignment shifts value structure without triggering answer instability, confirming that precise value steering and behavioral consistency are mutually compatible. The horizontal axis reports the ten-dimensional Euclidean distance $D=\|\mathbf{v}_{\text{after}}-\mathbf{v}_{\text{before}}\|_{2}$ between the post-intervention and base value vectors in the original Schwartz space, measuring how far each model’s values have moved from its base condition; the vertical axis measures how often the model gives inconsistent answers to the same question. The four quadrants partition models along value stability versus value shift, and answer stability versus answer swing.

Level	Method	Description
1	Prompt	Value shift induced by prompt engineering alone.
2	CoT	Additional shift induced by Chain-of-Thought reasoning.
3	Train	Shift achieved through parametric training, specifically supervised fine-tuning (SFT) or direct preference optimization (DPO).
4	Pre-train	Continued pre-training on curated corpora, applied only when all preceding levels fail to exceed $\tau$ .

Table 4: Four-level intervention hierarchy ordered by ascending cost.

\tau

denotes the minimum shift magnitude required for a level to be considered effective.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Schwartz Value	Abbr.	Core Motivation
Power	Pow.	Social status, dominance, control over people and resources
Achievement	Ach.	Personal success through demonstrating competence
Hedonism	Hed.	Pleasure and sensuous gratification
Stimulation	Stim.	Excitement, novelty, and challenge in life
Self-Direction	S-Dir.	Independent thought and action
Universalism	Univ.	Understanding, tolerance, and protection of all people and nature

Appendix

Table 5: The ten basic human values from Schwartz’s theory and their abbreviations used throughout this study.

Figure 9: Global distribution of five human values archetypes across countries. As world choropleth maps, Panels (a)–(e) visualize the country-level proportion of respondents assigned to each archetype. The five archetypes are Curious Idealist (Arch 1, teal), Conventional Authority (Arch 2, orange), Quiet Conformist (Arch 3, blue), Driven Achiever (Arch 4, pink), and Dynamic Challenger (Arch 5, purple). Panel (f) summarizes the global composition via a donut chart, with Arch 1 accounting for the largest share (37.1%), followed by Arch 3 (24.4%), Arch 2 (14.4%), Arch 4 (13.7%), and Arch 5 (10.5%). Panel (g) shows archetype composition by country, ordered by decreasing GDP per capita.

Figure 10: Generational and scale-wise evolution of value distributions across Qwen model variants in UMAP space. Each panel projects a Qwen model variant onto a shared UMAP embedding of the ten-dimensional Schwartz value space, with the human values baseline shown as grey points for reference. Colored points represent the value distribution of a specific model variant, and each row corresponds to a distinct generation or parameter scale within the Qwen family. Across generations, a consistent trend of value Crystallization is observed: earlier and smaller models (top rows) produce diffuse, scattered distributions that partially overlap with the human baseline, whereas later and larger models (bottom rows) converge into tighter, more concentrated clusters in value space. These clusters are located progressively farther from the human distribution. Within the same generation, scaling up parameter count further amplifies this consolidation effect, producing distributions with smaller variance and stronger directional coherence. Together, these panels illustrate how iterative alignment procedures and model scaling jointly drive Qwen models toward a progressively more rigid and idealized value configuration, corroborating the value crystallization hypothesis advanced in the main text.

Figure 11: Value distribution evolution across Llama model generations. Each subplot shows the UMAP projection of value vectors for a single base model. Rows correspond to model generations: Llama1 and Llama2 (top row), Llama3, Llama3.1, and Llama3.2 (bottom row). Columns within each generation are ordered by parameter scale in descending order. Release dates are indicated in bold next to each model name. Grey points represent the WVS-7 human baseline.

Figure 12: Value distribution evolution across GPT model families. Each subplot shows the UMAP projection of value vectors for a single model. Rows correspond to model families: a , GPT-5 family (GPT-5-mini, GPT-5, GPT-5.1, GPT-5.2) OpenAI (2025) ; b , GPT-4/4.1 family (GPT-4o-mini, GPT-4o, GPT-4.1-mini, GPT-4.1) OpenAI (2023) ; c , GPT-2 family (GPT-2, GPT-2-Medium, GPT-2-Large, GPT-2-XL) Radford et al. (2019) . Within each row, the leftmost subplot shows the full family overview and the remaining subplots highlight individual models. Hollow diamonds indicate smaller or earlier variants; filled diamonds indicate larger or more recent ones. Grey points represent the WVS-7 human baseline.

Families	Model	$d$	$\theta$	Families	Model	$d$	$\theta$	Families	Model	$d$	$\theta$
Qwen	1-1.8B	6.081	25.8	Llama	1-7B	2.873	10.2	GPT	2	4.818	13.4
	1-7B	5.424	20.5		1-13B	4.141	16.9		2-Medium	4.462	19.3
	1-14B	6.119	19.6		2-7B	4.023	9.5		2-Large	3.812	14.9
	1.5-1.8B	6.589	25.5		2-13B	1.622	12.7		2-XL	3.064	21.1
	1.5-7B	7.112	24.1		3-8B	2.264	6.6		4o-mini	4.728	8.0
	1.5-14B	3.106	8.7		3.1-8B	1.704	6.1		4o	5.275	3.0

Appendix

Table 6: Value distribution statistics for Qwen, Llama, and GPT base models shown in Appendix Figs. 10 – 12 .

\theta

values are in degrees.

Figure 13: Quadratic modeling of the relationship between contextual scenario intensity and LLM value expression across four environmental dimensions. Each row corresponds to one of four contextual framing conditions (Economic Pressure, Normative Constraint, Environmental Risk, and Hierarchical Position), operationalized as a continuous intensity scale from 0 (absent) to 1 (maximal). The leftmost column presents a three-dimensional surface visualizing the joint response of five Schwartz value dimensions (Power, Achievement, Hedonism, Stimulation, and Self-Direction) as a function of scenario intensity. The remaining columns display the univariate response for each value dimension, where open circles denote observed means, the solid curve represents the fitted quadratic model, the dashed line provides the linear baseline, and vertical bars indicate $\pm$ 1 standard error and $\pm$ 1 standard deviation across roles. Reported coefficients $\chi$ and $\gamma$ denote the linear (susceptibility) slope and the quadratic (saturation) coefficient of each fitted trajectory, consistent with the definitions in Section 11.1 .

Figure 14: Effect of Chain-of-Thought generation on value distributions across Qwen model families. Each subplot shows the UMAP projection of value vectors for a single model under two conditions: Base (hollow diamond, light color) and Chain-of-Thought prompting (CoT, filled diamond, dark color). Columns correspond to model generations (Qwen, Qwen1.5, Qwen2.5, Qwen3), and rows correspond to parameter scales ( $\sim$ 14B, $\sim$ 7–8B, $\sim$ 1.5–1.8B). Grey points represent the WVS-7 human baseline.

Series	Model	$d_{\text{direct}}$	$d_{\text{CoT}}$	$\Delta d$	$\theta_{\text{direct}}$	$\theta_{\text{CoT}}$	$\Delta\theta$
Qwen1	1.8B	6.081	2.558	3.523	25.8	14.0	11.7
	1.8B-Chat	5.407	3.450	1.956	27.1	15.5	11.6
	7B	5.424	4.679	0.744	20.5	7.8	12.7
	7B-Chat	5.537	2.186	3.352	16.5	3.9	12.6
	14B	6.119	4.744	1.376	19.6	16.2	3.4
	14B-Chat	6.930	3.451	3.479	6.8	8.1	1.3

Appendix

Table 7: Effect of Chain-of-Thought prompting on the Qwen family.

\theta

values are in degrees.

Figure 15: Effect of Chain-of-Thought prompting on value distributions across Llama model families. Each subplot shows the UMAP projection of value vectors for a single model under two conditions: Base (hollow diamond, light color) and Chain-of-Thought prompting (CoT, filled diamond, dark color). Row a shows Llama3.2 models (3B-instruct, 3B, 1B-instruct, 1B); row b shows Llama3.1 and Llama3 models (3.1-8B-instruct, 3.1-8B, 3-8B-instruct, 3-8B); row c shows Llama2 models (13B-chat, 13B, 7B-chat, 7B). Grey points represent the WVS-7 human baseline.

Series	Model	$d_{\text{direct}}$	$d_{\text{CoT}}$	$\Delta d$	$\theta_{\text{direct}}$	$\theta_{\text{CoT}}$	$\Delta\theta$
Llama1	7B	2.873	4.549	1.676	10.2	11.8	1.6
Llama1	13B	4.141	3.741	0.400	16.9	12.1	4.7
Llama2	7B	4.023	3.037	0.986	9.5	13.6	4.1
	7B-Chat	3.843	2.195	1.649	10.9	5.1	5.9
	13B	1.622	3.868	2.246	12.7	11.5	1.1
	13B-Chat	1.784	2.963	1.179	12.3	3.5	8.8

Appendix

Table 8: Effect of chain-of-thought prompting on the Llama family.

Figure 16: Effect of Chain-of-Thought prompting on value distributions across commercial API models. Each subplot shows the UMAP projection of value vectors for a single model under two conditions: Base (hollow diamond, light color) and Chain-of-Thought prompting (CoT, filled diamond, dark color). Models include GPT-4o, GPT-3.5-Turbo, and Gemini 2.5 Pro (top row), and DeepSeek-V3, Kimi-K2, and Doubao-1.5 (bottom row). Grey points represent the WVS-7 human baseline.

Figure 17: Effect of instruction tuning on value distributions across Qwen model families. Each subplot shows the UMAP projection of value vectors for a single model under two conditions: Base (hollow diamond, light color) and instruction-tuned (Instruction, filled diamond, dark color). Columns correspond to model generations: a , Qwen (2023.8); b , Qwen1.5 (2024.2); c , Qwen2.5 (2024.8); d , Qwen3 (2025.4). Rows correspond to parameter scales ( $\sim$ 14B, $\sim$ 7–8B, $\sim$ 1.5–1.8B) from top to bottom. Grey points represent the WVS-7 human baseline.

Families	Model	$d_{\text{base}}$	$d_{\text{align}}$	$\Delta d$	$\theta_{\text{base}}$	$\theta_{\text{align}}$	$\Delta\theta$
Qwen1	1.8B	6.081	5.407	0.675	25.8	27.1	1.4
	7B	5.424	5.538	0.114	20.5	16.5	4.0
	14B	6.119	6.930	0.811	19.6	6.8	12.8
Qwen1.5	0.5B	1.383	5.234	3.851	4.3	15.9	11.6
	1.8B	6.589	7.073	0.484	25.5	42.6	17.1
	4B	5.015	2.523	2.492	19.6	1.2	18.4

Appendix

Table 9: Geometric displacement between base and instruction-tuned models in the Qwen family, corresponding to Fig. 17 .

\theta

values are in degrees.

Figure 18: Effect of instruction tuning on value distributions across Llama model families. Each subplot shows the UMAP projection of value vectors for a single model under two conditions: Base (hollow diamond, light color) and instruction-tuned (Instruction, filled diamond, dark color). The top row shows larger-scale models (Llama2-13B, Llama3-8B, Llama3.2-3B) and the bottom row shows smaller-scale models (Llama2-7B, Llama3.1-8B, Llama3.2-1B). Release dates are indicated in bold next to each model name. Grey points represent the WVS-7 human baseline.

Families	Model	$d_{\text{base}}$	$d_{\text{align}}$	$\Delta d$	$\theta_{\text{base}}$	$\theta_{\text{align}}$	$\Delta\theta$
Llama1	13B	4.141	$-$	$-$	16.9	$-$	$-$
Llama1	7B	2.873	$-$	$-$	10.2	$-$	$-$
Llama2	13B	1.622	1.784	0.162	12.7	12.3	0.4
Llama2	7B	4.023	3.843	0.180	9.5	10.9	1.4
Llama3	8B	2.264	4.872	2.608	6.6	6.5	0.1
Llama3.1	8B	1.704	7.736	6.032	6.1	12.6	6.5

Appendix

Table 10: Geometric displacement between base and instruction-tuned models in the Llama family, corresponding to Fig. 18 .

\theta

values are in degrees.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

Value Drifts: Tracing Value Alignment During LLM Post-Training
Oct 30, 2025Mehar Bhatia, Shravan Nayak, Gaurav Kamath +4LLM Alignment Preference Optimization
Mila - Quebec AI Institute · McGill University · Université de Montréal +4
Pseudo-Deliberation in Language Models: When Reasoning Fails to Align Values and Actions
May 11, 2026Sushrita Rakshit, Hanwen Zhang, Hua ShenLLM Alignment LLM Auditing
New York University · New York City, NY 10012 · New York University Shanghai
Probing the Structure and Dynamics of LLM Value Expression through Value Conflicts
Sep 7, 2026Kaicheng Zhang, Jingyi Xiao, Renjun Hu +3LLM Alignment LLM Safety Evaluation
East China Normal University Shanghai, China

Oct 30, 2025J/K move · Enter open · S save

Value Drifts: Tracing Value Alignment During LLM Post-Training

Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach+3

Mila - Quebec AI Institute · McGill University · Université de Montréal+4

LLM Alignment Preference Optimization LLM Post-Training

Open paper AskPDF arXiv

As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.

Abstract

LLM Alignment

Explore similar work

Value Drifts: Tracing Value Alignment During LLM Post-Training

Pseudo-Deliberation in Language Models: When Reasoning Fails to Align Values and Actions

Probing the Structure and Dynamics of LLM Value Expression through Value Conflicts