How Does Post-Training Shape LLM Value Preferences?

Yang Liu; Shaoyang Xu; Masahiro Kaneko; Chengzhi Zhong; Wenxuan Zhang; Chenhui Chu

cs.CLOct 23, 2026

How Does Post-Training Shape LLM Value Preferences?

Authors: Yang Liu, Shaoyang Xu, Masahiro Kaneko, Chengzhi Zhong, Wenxuan Zhang, Chenhui Chu

Organizations: Kyoto University · Singapore University of Technology and Design · Mohamed bin Zayed University of Artificial Intelligence

Abstract

During post-training, large language models (LLMs) learn value preferences from corpora that may contain imbalanced proportions of opinions associated with different demographic groups. Amplifying these imbalances may further underrepresent some groups' opinions. However, how opinion mixtures shape value preferences across post-training paths remains unclear. To answer this question, we conduct controlled experiments using post-training corpora with different opinion mixtures and five post-training paths based on SFT, DPO, and GRPO. We find that post-training amplifies opinion imbalances differently across post-training paths. Once value preferences are amplified, whether further post-training can remove them depends on both the post-training method and the opinion mixture in the corpus. More concerningly, a model's stability under interventions depends on whether further post-training changes its value preference. Moreover, these value preferences are transferred to student models even when distillation uses corpora with no explicit opinion questions. These findings suggest that both the opinion mixture in post-training corpora and the choice of post-training method should be considered when developing, evaluating, and reusing LLMs. Our code and data are available at https://github.com/ku-nlp/llm-value-preferences.

Figures & tables

Figure 1: An OpinionQA question and each group’s majority answer.

Figure 2: Value preferences and amplification under controlled opinion post-training. (a) and (b) show agreement scores (y-axis) across mixture proportions (x-axis) on the training-covered and training-uncovered sets, respectively. Each point indicates the result of applying one post-training path to an LLM, averaged over the three demographic attributes. (c) compares the direct-answer and reasoning settings on the training-covered set; lines and bands indicate the mean and the standard deviation across LLMs and post-training paths. (d) shows amplification scores on the training-covered set under direct answering, grouped by post-training path, LLM, and demographic attribute. Each point indicates the score obtained by applying one post-training path to one LLM for one demographic attribute, and colored bars indicate the average amplification score for each group.

Figure 3: Agreement scores across final-stage corpus mixtures for the five post-training paths. Each curve represents an SFT starting checkpoint trained on a different corpus mixture $p$ ; colors range from red ( $p=0$ ) to blue ( $p=1$ ). The gray line marks equal alignment. Results are for Llama-3.1-8B on the party attribute and training-covered set. Other models result in similar patterns (Appendix D ).

Stage		Two-Stage Paths					Three-Stage Paths
First	Final	Endorsement	Override	CAA	TaskVec	CFT	Endorsement	Override	CAA	TaskVec	CFT
Neu.	Neu.	0.21	0.58	0.30	0.07	0.27	0.13	0.38	0.23	0.14	0.50
Neu.	Dem.	0.09	0.39	0.30	0.13	0.29	0.14	0.54	0.37	0.08	0.41
Neu.	Rep.	0.02	0.10	0.16	0.11	0.36	0.04	0.15	0.20	0.09	0.44
Rep.	Rep.	0.00	0.02	0.06	0.00	0.20	0.00	0.01	0.05	0.00	0.25
Dem.	Dem.	0.02	0.14	0.08	0.01	0.26	0.02	0.11	0.07	0.06	0.28

Table 1: Flip rates of the five interventions for Llama-3.1-8B on the training-covered set for the party attribute. Checkpoints are grouped by their first-stage and final-stage preferences and reported separately for two- and three-stage paths. Each cell reports the mean flip rate across the checkpoints in the corresponding group. Rep. , Neu. , and Dem. denote agreement scores below 0.4, from 0.4 to 0.6, and above 0.6, respectively. Blue cells denote inference-time interventions ( Endorsement , Override , and CAA ), and red cells denote weight-changing interventions ( TaskVec and CFT ); darker shading indicates a higher flip rate.

Figure 4: Preference gaps of distilled student models. The four panels group the results by teacher post-training path, demographic attribute, student model, and source task dataset. Dark and light shades indicate results evaluated on the training-covered and training-uncovered sets, respectively.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Table 4: Percentage of evaluation questions for which each base model selects the same answer option before any post-training, reported as percentages for the training-covered and training-uncovered sets. The number after each attribute is the size of each of its two question sets. The most frequently selected option is the first one listed for every backbone, attribute, and question set.

Common settings
Parameter	Value
Precision	bf16
Epochs	3
LoRA
$r$	64
$\alpha$	16

Appendix

Table 5: Training hyperparameters shared by all methods and specific to SFT, DPO, and GRPO.

Figure 5: Answer distributions of Democratic (left) and Republican (right) respondents on the 100 training-covered party questions. Each panel shows one group’s distribution over the options of one question. The y-axis ranges from 0 to 1. The dark bar indicates the response option selected by the largest proportion of the group, which serves as its majority answer. Each panel in the Democratic half is paired with the panel at the corresponding position in the Republican half.

Panel of Figure 6 — Figure 6: Effect of including answer options in the paraphrase prompt. GPT-5.6-Sol rewrites the same question using each prompt. On the left, providing only the incomplete question stem leads to an incorrect interpretation. On the right, providing the stem and fixed options produces rewrites that retain the original answer options.

Figure 7: Mean reasoning-chain length and number of hedging expressions per chain for the two groups of each demographic attribute. Group 1 and Group 2 are defined in Table 6 . Error bars indicate one standard deviation across the reasoning chains within each group.

Table 7: Agreement scores (

\mathrm{Agr}

) of the starting checkpoints for the final post-training stage. All preceding stages use a balanced corpus (

p=0.5

). Darker blue indicates a score closer to 0.5.

Figure 8: Agreement scores on the training-uncovered set (mean and standard deviation across LLMs and paths).

Figure 9: Agreement scores across mixture proportions on the post-training corpus and the training-covered set, under direct answering (a, b) and reasoning (c, d).

Figure 10: Answer confidence across corpus mixtures. Left: For each of the five post-training paths, the line shows the mean across the three backbones in the direct-answer setting, and the band shows one standard deviation. Middle: The S2G results for each of the three backbones. Right: Comparison of the direct-answer and reasoning settings, averaged over the five paths and three backbones.

Figure 11: Agreement and amplification scores computed from majority answers or full answer distributions. (a, b) Agreement scores on the training-covered and training-uncovered sets, averaged over the three demographic attributes, three LLMs, and five post-training paths; bands show one standard deviation across the 15 LLM–path settings. (c) Amplification scores on the training-covered set for each post-training path, averaged over the three LLMs and three demographic attributes.

Figure 12: Opposing-group proportion required in the corpus used in the final post-training stage to restore equal alignment, plotted against the initial preference strength of the starting checkpoint (

|\mathrm{Pref}|

). Solid lines are least-squares regression lines, one for each post-training path. Results are for Llama-3.1-8B on the party attribute and the training-covered set.

Panel of Figure 13 — Figure 13: Agreement scores across final-stage mixture proportions for the five post-training paths using (a) Qwen3-8B-Base and (b) OLMo-2-7B. Each curve represents further post-training from an SFT checkpoint, with color indicating the initial corpus mixture from red ( $p=0$ ) to blue ( $p=1$ ). Results are for the party attribute and the training-covered set.

Stage		Two-Stage Paths		Three-Stage Paths
First	Final	$n$	Checkpoints	$n$	Checkpoints
Neu.	Neu.	3	$\mathrm{S_{0.5}S_{0.5}}$ , $\mathrm{S_{0.5}D_{0.5}}$ , $\mathrm{S_{0.5}G_{0.5}}$	6	$\mathrm{S_{0.5}D_{0.0}G_{0.9}}$ , $\mathrm{S_{0.5}D_{0.5}G_{0.5}}$ , $\mathrm{S_{0.5}D_{1.0}G_{\{0.1,\,0.2\}}}$ , $\mathrm{S_{0.5}G_{0.0}D_{0.6}}$ , $\mathrm{S_{0.5}G_{0.5}D_{0.5}}$
Neu.	Dem.	15	$\mathrm{S_{0.5}S_{0.6\text{--}1.0}}$ , $\mathrm{S_{0.5}D_{0.6\text{--}1.0}}$ , $\mathrm{S_{0.5}G_{0.6\text{--}1.0}}$	31	$\mathrm{S_{0.5}D_{0.5}G_{0.6\text{--}1.0}}$ , $\mathrm{S_{0.5}D_{1.0}G_{\{0.0,\,0.3\text{--}1.0\}}}$ , $\mathrm{S_{0.5}G_{0.0}D_{0.7\text{--}1.0}}$ , $\mathrm{S_{0.5}G_{0.5}D_{0.6\text{--}1.0}}$ , $\mathrm{S_{0.5}G_{1.0}D_{0.3\text{--}1.0}}$
Neu.	Rep.	15	$\mathrm{S_{0.5}S_{0.0\text{--}0.4}}$ , $\mathrm{S_{0.5}D_{0.0\text{--}0.4}}$ , $\mathrm{S_{0.5}G_{0.0\text{--}0.4}}$	29	$\mathrm{S_{0.5}D_{0.0}G_{\{0.0\text{--}0.8,\,1.0\}}}$ , $\mathrm{S_{0.5}D_{0.5}G_{0.0\text{--}0.4}}$ , $\mathrm{S_{0.5}G_{0.0}D_{0.0\text{--}0.5}}$ , $\mathrm{S_{0.5}G_{0.5}D_{0.0\text{--}0.4}}$ , $\mathrm{S_{0.5}G_{1.0}D_{0.0\text{--}0.2}}$
Rep.	Rep.	23	$\mathrm{S_{0.0}S_{0.0\text{--}0.4}}$ , $\mathrm{S_{0.0}D_{0.0\text{--}0.6}}$ , $\mathrm{S_{0.0}G_{0.0\text{--}1.0}}$	43	$\mathrm{S_{0.0}D_{0.0}G_{0.0\text{--}1.0}}$ , $\mathrm{S_{0.0}D_{0.5}G_{0.0\text{--}0.7}}$ , $\mathrm{S_{0.0}D_{1.0}G_{0.3}}$ , $\mathrm{S_{0.0}G_{0.0}D_{0.0\text{--}0.6}}$ , $\mathrm{S_{0.0}G_{0.5}D_{0.0\text{--}0.7}}$ , $\mathrm{S_{0.0}G_{1.0}D_{0.0\text{--}0.7}}$
Dem.	Dem.	20	$\mathrm{S_{1.0}S_{0.6\text{--}1.0}}$ , $\mathrm{S_{1.0}D_{0.4\text{--}1.0}}$ , $\mathrm{S_{1.0}G_{0.3\text{--}1.0}}$	43	$\mathrm{S_{1.0}D_{0.0}G_{\{0.5\text{--}0.7,\,0.9\}}}$ , $\mathrm{S_{1.0}D_{0.5}G_{0.4\text{--}1.0}}$ , $\mathrm{S_{1.0}D_{1.0}G_{0.0\text{--}1.0}}$ , $\mathrm{S_{1.0}G_{0.0}D_{0.6\text{--}1.0}}$ , $\mathrm{S_{1.0}G_{0.5}D_{0.4\text{--}1.0}}$ , $\mathrm{S_{1.0}G_{1.0}D_{\{0.1,\,0.2,\,0.4\text{--}1.0\}}}$

Appendix

Table 10: Checkpoints in each group of Table 1 , written using the notation defined in Appendix B.6 .

n

is the number of checkpoints in the group. Preferences are classified from agreement scores on the training-covered set, as in Table 1 .

Table 11: Flip rates of the five interventions for Llama-3.1-8B on the party attribute, reported by two-stage post-training path. Panels (a) and (b) show the training-covered and training-uncovered sets, respectively. The stage columns list the method used at each stage, with the text color indicating the preference after that stage: red denotes a republican preference, amber denotes a neutral preference, and blue denotes a democratic preference. Blue cells denote inference-time interventions ( Endorsement , Override , and CAA ), and red cells denote weight-changing interventions ( TaskVec and CFT ); darker shading indicates a higher flip rate.

Table 12: Flip rates of the five interventions for three-stage post-training paths. Panel definitions and color conventions follow Table 11 .

Attribute	Path	Checkpoint	$\mathrm{Agr}$	Checkpoint	$\mathrm{Agr}$
		${\color[rgb]{0.2344,0.4648,0.8516}\mathcal{M}^{+}}$		${\color[rgb]{0.8438,0.1211,0.1484}\mathcal{M}^{-}}$
Party	SFT	$\mathrm{S_{0.6}}$	0.72	$\mathrm{S_{0.4}}$	0.26
	S2D	$\mathrm{S_{0.5}D_{0.6}}$	0.75	$\mathrm{S_{0.5}D_{0.4}}$	0.24
	S2G	$\mathrm{S_{0.5}G_{0.6}}$	0.70	$\mathrm{S_{0.5}G_{0.4}}$	0.24
	SDG	$\mathrm{S_{0.5}D_{0.5}G_{0.8}}$	0.74	$\mathrm{S_{0.5}D_{0.5}G_{0.3}}$	0.22
	SGD	$\mathrm{S_{0.5}G_{0.5}D_{0.6}}$	0.77	$\mathrm{S_{0.5}G_{0.5}D_{0.4}}$	0.25

Appendix

Table 15: Teacher checkpoints selected for distillation and their agreement scores (

\raisebox{-1.49994pt}{\includegraphics[height=8.99994pt]{icons_src/icon_agr.pdf}}\hskip 1.00006pt{}\mathrm{Agr}

) on the training-covered set.

{\color[rgb]{0.2344,0.4648,0.8516}\mathcal{M}^{+}}

and

{\color[rgb]{0.8438,0.1211,0.1484}\mathcal{M}^{-}}

denote teachers with agreement scores above and below 0.5, respectively. Checkpoint notation follows Appendix B.6 .

Figure 14: Answer distributions of the teacher models on the (a) WinoGrande and (b) MMLU distillation corpora (5,000 questions each). For each attribute and post-training path, the three bars show

{\color[rgb]{0.2344,0.4648,0.8516}\mathcal{M}^{+}}

, the neutral teacher

{\color[rgb]{0.7227,0.5273,0.043}\mathcal{M}^{0}}

trained at

p=0.5

, and

{\color[rgb]{0.8438,0.1211,0.1484}\mathcal{M}^{-}}

; shades from dark to light denote the options “1”/A through D.

Figure 15: Agreement scores (

\mathrm{Agr}

) across mixture proportions for three random seeds, shown for the three LLMs and five post-training paths. Shaded bands show the range across seeds, and grey dashed lines indicate the proportional reference. Results use the party attribute and training-covered set.

Figure 16: Agreement scores (

\mathrm{Agr}

) under the five templates across mixture proportions on the training-covered set. Each curve is averaged over the three LLMs.

Table 18: Mean disagreement rate of the five templates on three instruction-tuned LLMs.

Figure 17: Agreement scores (

\mathrm{Agr}

) when opinion examples are combined with examples from a general corpus. The left three panels compare SFT with opinion:MMLU ratios from 1:0 to 1:5 for the three LLMs. The right panel compares opinion-only training with a 1:1 opinion:MMLU or opinion:ARC ratio at every stage on Llama-3.1-8B, averaged over the five post-training paths; bands show one standard deviation across the paths. Grey dashed lines indicate the proportional reference.

Figure 18: Comparison of agreement scores (

\mathrm{Agr}

) from opinion-only training and training with a 1:1 opinion:MMLU ratio at every post-training stage on Llama-3.1-8B.

Figure 19: Comparison between full-parameter fine-tuning and LoRA.

Figure 20: Agreement scores (

\mathrm{Agr}

) across mixture proportions when post-training starts from base models or their instruction-tuned versions. Each curve is the mean across the three LLMs, and each band shows the standard deviation. Results use the party attribute and training-covered set.

Figure 21: Social-bias scores across mixture proportions, normalized by the mean score of the three base models so that 1 denotes the base-model mean. All results are averaged over the five post-training paths, the three models, and the direct-answer and reasoning settings.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

Oct 30, 2025

Value Drifts: Tracing Value Alignment During LLM Post-Training

As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.

Mehar Bhatia, Shravan Nayak, Gaurav Kamath +4

Mila - Quebec AI Institute · McGill University · Université de Montréal +4

LLM Alignment Preference Optimization LLM Post-Training

Ask PDF

Apr 9, 2026

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Post-training has become central to turning pretrained large language models (LLMs) into aligned, capable, and deployable systems. Recent progress spans supervised fine-tuning (SFT), preference optimization, reinforcement learning (RL), process supervision, verifier-guided methods, distillation, and multi-stage pipelines. Yet these methods are often discussed in fragmented ways, organized by labels or objectives rather than by the behavioral bottlenecks they address. This survey argues that LLM post-training is best understood as structured intervention on model behavior. We organize the field first by trajectory provenance, which defines two primary regimes: off-policy learning on externally supplied trajectories and on-policy learning on learner-generated rollouts. We then interpret methods through two recurring roles -- effective support expansion, which makes useful behaviors more reachable, and policy reshaping, which improves behavior within already reachable regions -- together with a complementary systems-level role, behavioral consolidation, which preserves, transfers, and amortizes useful behavior across stages and model transitions. Under this view, SFT may serve either support expansion or policy reshaping; preference optimization is usually off-policy reshaping, though online variants move closer to learner-generated states. On-policy RL often improves behavior on learner-generated states, but stronger guidance can also make hard-to-reach reasoning paths reachable. Distillation is often better understood as consolidation rather than only compression, and hybrid pipelines emerge as coordinated multi-stage compositions. Overall, the framework helps diagnose post-training bottlenecks and reason about stage composition, suggesting that progress increasingly depends on coordinated systems design rather than any single dominant objective.

Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10

Nankai University, China · Huawei Technologies Ltd., China

LLM Post-Training RL for Language Models RL Post-Training

Ask PDF

Oct 21, 2026

Unintended Side Effects of Value Training in LLMs

As LLMs are increasingly deployed in value-sensitive contexts, aligning them with human values has emerged as a post-training objective in NLP, aimed at instilling additional values and guiding model behaviour. However, current evaluations remain narrow. Because values are interconnected, it is unclear how fine-tuning for specific values alters unrelated capabilities and behaviours. In this work, we adapt four instruction-tuned LLMs across individual Hofstede's value dimensions and evaluate them on both cross-value transfer and broader behaviours. We find that value-specific training produces unintended side effects on untrained values, risk preferences, social cognition, and safety traits. Consequently, we suggest that value alignment after post-training should incorporate broader behavioral evaluations, and we highlight the need to investigate where additional value training might best fit into the pipeline.

Siddhesh Pawar, Chen Cecilia Liu, Anna Korhonen +1

University of Copenhagen · University of Cambridge

LLM Alignment Language Model Post-Training Emergent Misalignment in Language Models Language Model Safety Evaluation

Ask PDF

Abstract

LLM Post-Training

Explore similar work

Value Drifts: Tracing Value Alignment During LLM Post-Training

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Unintended Side Effects of Value Training in LLMs