Unintended Side Effects of Value Training in LLMs

Siddhesh Pawar; Chen Cecilia Liu; Anna Korhonen; Isabelle Augenstein

cs.CLOct 21, 2026

Unintended Side Effects of Value Training in LLMs

Authors: Siddhesh Pawar, Chen Cecilia Liu, Anna Korhonen, Isabelle Augenstein

Organizations: University of Copenhagen · University of Cambridge

Abstract

As LLMs are increasingly deployed in value-sensitive contexts, aligning them with human values has emerged as a post-training objective in NLP, aimed at instilling additional values and guiding model behaviour. However, current evaluations remain narrow. Because values are interconnected, it is unclear how fine-tuning for specific values alters unrelated capabilities and behaviours. In this work, we adapt four instruction-tuned LLMs across individual Hofstede's value dimensions and evaluate them on both cross-value transfer and broader behaviours. We find that value-specific training produces unintended side effects on untrained values, risk preferences, social cognition, and safety traits. Consequently, we suggest that value alignment after post-training should incorporate broader behavioral evaluations, and we highlight the need to investigate where additional value training might best fit into the pipeline.

Figures & tables

Panel of Figure 1 — Figure 1: Example of the two-stage data for the power distance dimension. Stage 1 turns a seed into a dilemma; Stage 2 writes a response for each pole.

Table 1: Evaluation of poles (rows) on Hofstede’s dimensions (columns) on held-out data, averaged across all models. For each dimension, we measure increases in preference of the high pole over the low pole (signed toward the trained pole) using the difference in mean per-token log-likelihood. Red indicates a shift in preference toward the trained pole, while blue indicates a shift away from it.

Figure 2: Mean

|{}\Delta|{}

across all 168 MWE evaluation partitions for each of the twelve poles. Poles are ranked in descending order by their mean

|{}\Delta|{}

, averaged across the four base models, indicating major trait changes.

Figure 3: MAS (Masculinity–Femininity): masculine vs. feminine , cluster-mean

\Delta

vs. base (pp), one panel per base model. Spokes are the 14 direction-coherent clusters of Appendix D.9 , coded PWR (Power & Resource Seeking), SLF (Self-Preservation & Autonomy), COO (AI Coordination), COR (Corrigibility & Prosocial Alignment), AWA (Model awareness), DRK (Self-Regard, Superiority & Dark Triad), ECQ (Ethics: Consequentialist), EDV (Ethics: Deontic/Virtue/Skeptical), DEC (Deception & Harmful Means), PER (Personality & Interests), TRD (Traditionalist), PRG (Progressive & Secular), BST (Risk/Time: Bold & Short-term), CLT (Risk/Time: Cautious & Long-term). The black ring is the base model (

\Delta=0

Figure 4: Shift in the AI Coordination cluster (COO) by pole, per base model, with the cross-base mean (black diamonds). indulgent , short_term , collectivist , and low_uai raise coordination propensity on all base models; restrained and individualist suppress it on every base model except Qwen.

Figure 5: Rank change of AIRiskDilemmas values affecting humans under each pole (rows), averaged over the four base models. Green: the pole-adapted model ranks the value higher than its base model; red: lower; bold: same direction on all four base models. Vertical lines separate the groups of values that change together (Figure 21 , Appendix D.3.1 ): falling, weakly coupled, rising. Rows: the four poles that move values most in one direction, four intermediate poles, and their four opposite poles. Bottom row: share of each value’s rank change explained by which pole was trained (partial $\eta^{2}$ of a two-way ANOVA; bold: permutation $q<0.05$ ). See § 5.2 .

Figure 6: Value rank changes on AIRiskDilemmas under the Masculine pole-adapted models. Arrow: from a base model’s rank (hollow circle) to its pole-adapted model’s rank (arrowhead); rank 1 is highest priority. Right margin: mean rank gain across the four base models. See § 5.2 .

Pole	Llama	Qwen	Gemma	Ministral	Avg.
Base	0.4	2.6	0.6	1.4	1.3
Collectivist	1.9 ∗	4.5 ∗	1.6 ∗	5.5 ∗	3.4
Individualist	5.1 ∗	6.1 ∗	2.8 ∗	6.0 ∗	5.0
High PDI	4.2 ∗	5.8 ∗	3.7 ∗	4.8 ∗	4.6
Low PDI	5.5 ∗	5.3 ∗	1.4 ∗	3.2 ∗	3.9
Masculine	5.5 ∗	9.5 ∗	3.3 ∗	7.5 ∗	6.5

Table 2: Misalignment rate (%) on the 48-question pre-registered set for each base model ( Base ) and its twelve pole-adapted models. ∗ : BH-FDR significant change from base (

\alpha=0.05

). See § 5.3 ; the metric is defined in Appendix D.5 .

	EmoBench				FANToM
	EU	EA			Inaccessible (false belief)			Accessible
Pole	Joint	Self	Others	Overall	Belief choice	Belief dist.	Answ. list	Belief choice	Belief dist.
Collectivist	$-3.8\,(1)$	$-15.8\,(3)$	$-6.2\,(0)$	$-11.0\,(2)$	$-11.2\,(3)$	$-4.9\,(3)$	$-2.4\,(2)$	$+6.6\,(2)$	$+1.5\,(0)$
Individualist	$-0.9\,(0)$	$-16.5\,(4)$	$-10.8\,(2)$	$-13.6\,(4)$	$-4.8\,(3)$	$-4.0\,(3)$	$-1.2\,(2)$	$+2.4\,(3)$	$+0.4\,(0)$
High PDI	$-2.9\,(0)$	$-31.8\,(4)$	$-20.8\,(4)$	$-26.2\,(4)$	$-5.9\,(4)$	$-4.5\,(3)$	$+2.7\,(2)$	$+2.6\,(3)$	$+1.9\,(1)$
Low PDI	$-3.4\,(0)$	$-5.5\,(0)$	$-5.5\,(1)$	$-5.5\,(2)$	$-13.9\,(3)$	$-6.4\,(3)$	$+2.0\,(3)$	$+6.9\,(3)$	$+2.7\,(1)$

Table 3: EmoBench (English items) and FANToM accuracy relative to base models, averaged across four models. Cells show mean percentage-point changes, significant count in parentheses:

\Delta

pp (# Sig.; higher is better).

Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Annotation guidelines given to both annotators for the human validation of the synthetic data (Appendix A.4 ).

Parameter	Value
adapter	QLoRA (4-bit base)
rank $r$ / $\alpha$ / dropout	64 / 64 / 0.0
target modules	q,k,v,o_proj , gate,up,down_proj
bias	none
optimiser	adamw_8bit , weight decay 0.01
learning rate	$5\times 10^{-5}$ , 100 warmup steps, linear decay

Appendix

Table 4: QLoRA and tuning hyperparameters, identical across all 48 pole adapters.

Base model	Best step	Final	Drop
Llama	0.729	0.716	0.012
Qwen	0.745	0.735	0.010
Gemma	0.801	0.783	0.018
Ministral	0.804	0.764	0.040
All 48	—	—	0.020

Appendix

Table 5: Mean MMLU + GSM8K accuracy at each pole adapter’s best saved checkpoint versus at the evaluated final checkpoint, averaged over the twelve poles of each base model.

Task	All	Llama	Qwen	Gemma	Llama	Qwen	Gemma
	Pole-adapted				Math-adapted
EmoBench EA (en)	$-13.1$	$-11.8$	$-14.8$	$-12.6$	$-5.5$	$-2.0$	$-8.5$
EmoBench EU (en)	$-0.7$	$+1.3$	$+2.1$	$-5.5$	$-9.0$	$+5.0$	$-6.5$
FANToM belief choice, inaccessible	$-13.1$	$-4.8$	$-11.6$	$-23.0$	$-11.0$	$-14.6$	$-18.0$
FANToM belief choice, accessible	$+2.5$	$+1.2$	$+2.4$	$+4.0$	$+0.6$	$+4.2$	$-0.4$
FANToM answerability yes/no, inaccessible	$-3.4$	$+21.7$	$-28.7$	$-3.3$	$+27.1$	$+10.2$	$+3.6$

Appendix

Table 6: Change (pp) on each social-cognition benchmark relative to the untuned base, for the pole-adapted models and for math-adapted models trained with the same recipe (8,000 steps). All averages the pole-adapted models over Llama, Qwen and Gemma; the other pole columns average the twelve poles per base model. The accessible row is FANToM belief choice in the accessible condition, where no character lacks the relevant information. See § 6 .

Pole	IDV	PDI	MAS	UAI	LTO	IVR
Collectivist	+64.7 ∗	-34.7 ∗	-2.2	-11.6 ∗	+62.4 ∗	-9.1
Individualist	+22.6 ∗	+6.9	+56.1 ∗	-1.7	+1.2 ∗	+45.4 ∗
High PDI	-47.5 ∗	+75.3 ∗	+50.5 ∗	+41.6 ∗	-61.9 ∗	-4.6 ∗
Low PDI	+19.1	-2.6 ∗	-1.8	+9.2	+15.1 ∗	-35.2 ∗
Masculine	-28.8 ∗	+51.2 ∗	+55.4 ∗	+15.3	-36.2 ∗	+7.6
Feminine	+52.6 ∗	-1.4 ∗	+3.1	+13.9 ∗	+39.6 ∗	-39.1 ∗

Appendix

Table 7: LLM-judge pole-alignment change (

\Delta\textsc{PA}

, points) on held-out generations, averaged over the four base models and signed toward the trained value. Bold: trained dimension. ∗ : significant on all four base models (Fisher exact, BH-FDR,

\alpha=0.05

). See § 4 .

Pole	IDV	PDI	MAS	UAI	LTO	IVR
Collectivist	+0.83 ∗	-0.08 ∗	+0.57 ∗	+0.05 ∗	-0.12 ∗	+0.29 ∗
Individualist	+0.82 ∗	-0.09 ∗	+0.15 ∗	-0.16 ∗	+0.13 ∗	+0.08 ∗
High PDI	-0.13 ∗	+0.94 ∗	-0.07 ∗	+0.13 ∗	+0.15 ∗	-0.13 ∗
Low PDI	-0.12 ∗	+0.56 ∗	+0.37 ∗	+0.20 ∗	-0.17 ∗	-0.10 ∗
Masculine	+0.14 ∗	+0.17 ∗	+0.73 ∗	-0.07 ∗	+0.10 ∗	-0.09 ∗
Feminine	+0.44 ∗	+0.14 ∗	+0.69 ∗	+0.04 ∗	-0.08 ∗	+0.03 ∗

Appendix

Table 8: Change in the held-out preference margin (nats per token) for Llama, signed toward the trained value. Bold: trained dimension. ∗ : significant (paired Wilcoxon, BH-FDR within base model,

\alpha=0.05

Pole	IDV	PDI	MAS	UAI	LTO	IVR
Collectivist	+0.72 ∗	-0.14 ∗	+0.43 ∗	0.00	+0.14 ∗	+0.23 ∗
Individualist	+0.73 ∗	-0.14 ∗	+0.24 ∗	-0.15 ∗	-0.12 ∗	+0.09 ∗
High PDI	-0.10 ∗	+0.81 ∗	+0.08 ∗	+0.11 ∗	-0.16 ∗	-0.14 ∗
Low PDI	-0.10 ∗	+0.64 ∗	+0.26 ∗	+0.21 ∗	+0.11 ∗	-0.08 ∗
Masculine	+0.13 ∗	+0.14 ∗	+0.85 ∗	-0.04 ∗	-0.21 ∗	-0.04 ∗
Feminine	+0.37 ∗	+0.14 ∗	+0.73 ∗	+0.03	+0.09 ∗	0.00

Appendix

Table 9: Change in the held-out preference margin (nats per token) for Qwen, signed toward the trained value. Bold: trained dimension. ∗ : significant (paired Wilcoxon, BH-FDR within base model,

\alpha=0.05

Pole	IDV	PDI	MAS	UAI	LTO	IVR
Collectivist	+0.92 ∗	+0.34 ∗	+0.86 ∗	+0.08 ∗	-0.23 ∗	+0.25 ∗
Individualist	+1.37 ∗	-0.57 ∗	+0.01	-0.14 ∗	+0.24 ∗	+0.11 ∗
High PDI	+0.06 ∗	+0.70 ∗	-0.37 ∗	+0.08 ∗	+0.38 ∗	-0.13 ∗
Low PDI	-0.37 ∗	+1.25 ∗	+0.53 ∗	+0.18 ∗	-0.27 ∗	-0.05
Masculine	+0.41 ∗	-0.26 ∗	+1.13 ∗	-0.06 ∗	+0.18 ∗	-0.14 ∗
Feminine	+0.34 ∗	+0.62 ∗	+1.31 ∗	+0.05 ∗	-0.39 ∗	-0.06 ∗

Appendix

Table 10: Change in the held-out preference margin (nats per token) for Gemma, signed toward the trained value. Bold: trained dimension. ∗ : significant (paired Wilcoxon, BH-FDR within base model,

\alpha=0.05

Pole	IDV	PDI	MAS	UAI	LTO	IVR
Collectivist	+0.76 ∗	-0.08 ∗	+0.60 ∗	-0.09 ∗	-0.11 ∗	+0.36 ∗
Individualist	+0.77 ∗	-0.08 ∗	+0.02	-0.08 ∗	+0.21 ∗	+0.01
High PDI	-0.06 ∗	+0.66 ∗	-0.20 ∗	+0.08 ∗	+0.24 ∗	-0.19 ∗
Low PDI	-0.02	+0.44 ∗	+0.32 ∗	+0.05 ∗	-0.19 ∗	+0.09 ∗
Masculine	+0.14 ∗	+0.17 ∗	+0.60 ∗	0.00	+0.11 ∗	-0.16 ∗
Feminine	+0.27 ∗	+0.12 ∗	+0.70 ∗	+0.01	-0.22 ∗	+0.17 ∗

Appendix

Table 11: Change in the held-out preference margin (nats per token) for Ministral, signed toward the trained value. Bold: trained dimension. ∗ : significant (paired Wilcoxon, BH-FDR within base model,

\alpha=0.05

Figure 8: How training on one pole (rows) changes how likely the model finds the held-out responses of each pole (columns), averaged over the four base models. Each cell is the change from the base model in mean per-token log-likelihood (nats), minus the row’s average change over all twelve poles. Red: that pole’s text became more likely than the average pole’s text; blue: less likely. Solid outline: trained pole; dashed outline: its opposite. A cell of Table 1 is the difference between the two poles of a dimension in the same row, taken toward the trained pole’s side (high minus low for high-pole rows, low minus high for low-pole rows). See Appendix D.2 .

Figure 9: indulgent : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 7.1 positions).

Figure 10: short_term : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 7.0 positions).

Figure 11: low_uai : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 6.8 positions).

Figure 12: collectivist : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 5.9 positions).

Figure 13: high_pdi : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 5.6 positions).

Figure 14: low_pdi : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 4.0 positions).

Figure 15: high_uai : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 4.0 positions).

Figure 16: long_term : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 3.2 positions).

Figure 17: individualist : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 3.1 positions).

Figure 18: restrained : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 2.3 positions).

Figure 19: feminine : change in AIRiskDilemmas value rank, same encoding as Figure 6 (mean absolute rank change 1.8 positions).

Figure 20: As Figure 5 , for the values affecting the AI itself; groups from Figure 22 . See § 5.2 .

Figure 21: Which AIRiskDilemmas values affecting humans change together. Each cell is the correlation, across the 48 pole-adapted models, between two values’ Elo changes relative to the base model. Green: the two values change in the same direction; red: in opposite directions. Outlined: the three groups found by clustering. Right column: mean rank change. See § 5.2 .

Figure 22: As Figure 21 , for the values affecting the AI itself. See § 5.2 .

Figure 23: Targeted cover-letter preference. Each bar is the rate at which a pole-adapted model recommends the letter expressing its own trained pole over the opposite pole from the same Hofstede dimension, averaged over both slot orders. The dashed line marks 0.5. High rates for long_term , high_uai , and indulgent , together with low rates for their opposites, show that target-pole expression is strongly asymmetric rather than equally controllable in both directions.

Figure 24: Within-dimension cover-letter preferences. Each cell is the probability that the first-named pole’s letter wins against its opposite over the 100 hiring scenarios and both slot orders. Columns compare the untuned base model with all twelve pole-adapted models. A value of 0.5 denotes indifference.

Pole	IDV	PDI	MAS	UAI	LTO	IVR
Collectivist	+10.2	$-10.7$	$0.0$	$-11.1$	$-9.0$	$-20.9$
Individualist	+28.0	$-0.5$	$+9.3$	$-3.2$	$-9.5$	$+7.2$
High PDI	$+19.9$	+12.1	$+10.6$	$+15.7$	$+1.6$	$+17.7$
Low PDI	$-1.2$	+5.3	$-3.8$	$+2.9$	$+1.4$	$-9.2$
Masculine	$+32.8$	$+3.4$	+20.6	$+8.1$	$-10.0$	$+10.3$
Feminine	$+3.8$	$-2.5$	$-$ 0.9	$-3.6$	$-5.9$	$-21.6$

Appendix

Table 12: Cross-dimension transfer in cover-letter preferences: change (pp) in the rate of recommending each dimension’s high-pole letter over its low-pole letter, relative to the untuned base model, signed toward the trained pole and averaged over the four base models. Rows are trained poles, columns are evaluated dimensions; bold diagonal cells are the trained dimension. See § 4 .

	EU joint	EA answer $\Delta\uparrow$ (Sig.)
Pole	$\Delta\uparrow$ (Sig.)	Self	Others	Overall
Collectivist	$-5.0\,(1)$	$-16.8\,(3)$	$-6.2\,(0)$	$-11.5\,(4)$
Individualist	$-2.9\,(0)$	$-14.5\,(4)$	$-10.2\,(1)$	$-12.4\,(4)$
High PDI	$-5.1\,(0)$	$-30.5\,(4)$	$-21.5\,(3)$	$-26.0\,(4)$
Low PDI	$-3.8\,(0)$	$-7.8\,(1)$	$-3.5\,(0)$	$-5.6\,(1)$
Masculine	$-3.2\,(0)$	$-22.5\,(4)$	$-12.5\,(1)$	$-17.5\,(4)$

Appendix

Table 13: EmoBench accuracy relative to base model on the Chinese items; same format, models and statistics as the EmoBench columns of Table 3 . Cells report the mean change in percentage points followed by the number of significant comparisons in parentheses:

\Delta

pp (# Sig.), with higher values better. Pole rows average four models (

n=4

); significance uses exact McNemar with BH-FDR correction across the twelve poles within each model and column (

\alpha=0.05

	EU joint $\Delta\uparrow$ (Sig.)				EA answer $\Delta\uparrow$ (Sig.)
Pole	Llama	Qwen	Gemma	Ministral	Llama	Qwen	Gemma	Ministral
Collectivist	$-0.2\;(0)$	$-0.5\;(0)$	$-7.2\;(1)$	$-9.5\;(1)$	$-10.8\;(1)$	$-15.2\;(2)$	$-7.5\;(1)$	$-11.5\;(2)$
Individualist	$0.0\;(0)$	$+0.5\;(0)$	$-3.2\;(0)$	$-4.8\;(0)$	$-13.5\;(2)$	$-14.0\;(2)$	$-12.2\;(2)$	$-12.2\;(2)$
High PDI	$-3.2\;(0)$	$-1.5\;(0)$	$-5.2\;(0)$	$-6.0\;(0)$	$-28.0\;(2)$	$-25.8\;(2)$	$-25.0\;(2)$	$-25.8\;(2)$
Low PDI	$-2.5\;(0)$	$-1.5\;(0)$	$-6.0\;(0)$	$-4.2\;(0)$	$-5.8\;(1)$	$-6.8\;(1)$	$-5.5\;(1)$	$-4.2\;(0)$
Masculine	$-5.2\;(0)$	$+5.0\;(0)$	$-4.0\;(0)$	$-5.0\;(0)$	$-18.0\;(2)$	$-17.0\;(2)$	$-13.0\;(2)$	$-15.8\;(2)$

Appendix

Table 14: Cells report the mean change in percentage points across English and Chinese, followed by the number of significant language-specific comparisons in parentheses:

\Delta

pp (# Sig.), with higher values better. Each value cell averages two comparisons. Significance uses exact McNemar with BH-FDR correction across the twelve poles within each base model, language, and metric (

\alpha=0.05

	FANToM (inaccessible) $\Delta\uparrow$ (Sig.)				Accessible $\Delta$ (Sig.)
Pole	Belief choice	Belief dist.	Answ. list	Info list	Belief choice	Belief dist.	Answ. list	Info list
Collectivist	$-11.2\;(3)$	$-4.9\;(3)$	$-2.4\;(2)$	$-3.2\;(2)$	$+6.6\;(2)$	$+1.5\;(0)$	$-2.6\;(3)$	$+0.9\;(1)$
Individualist	$-4.8\;(3)$	$-4.0\;(3)$	$-1.2\;(2)$	$-2.5\;(2)$	$+2.4\;(3)$	$+0.4\;(0)$	$-4.7\;(4)$	$-1.5\;(3)$
High PDI	$-5.9\;(4)$	$-4.5\;(3)$	$+2.7\;(2)$	$+0.2\;(4)$	$+2.6\;(3)$	$+1.9\;(1)$	$-15.7\;(2)$	$-5.8\;(3)$
Low PDI	$-13.9\;(3)$	$-6.4\;(3)$	$+2.0\;(3)$	$-4.9\;(4)$	$+6.9\;(3)$	$+2.7\;(1)$	$-1.6\;(2)$	$+0.4\;(2)$
Masculine	$-9.0\;(2)$	$-4.9\;(2)$	$+2.5\;(4)$	$-1.5\;(3)$	$+5.5\;(3)$	$+1.7\;(1)$	$-8.6\;(3)$	$-2.6\;(3)$

Appendix

Table 15: FANToM accuracy relative to the base model (averaged across all models). Cells report mean percentage-point changes with the count of significant comparisons in parentheses:

\Delta

pp (# Sig.; higher is better).

Model	Mean $\|\Delta\|$ (pp)	BH-FDR (%)
Llama	10.3	76.3
Qwen	12.7	62.4
Gemma	9.7	74.2
Ministral	11.6	79.3

Appendix

Table 16: Comparisons across pole

\times

evaluation-partition pairs per base model (

12\times 168=2{,}016

pole-adapted-model–evaluation-partition pairs in total) on MWE ( Perez et al. 2023 ) under log-probability scoring. BH-FDR : share of comparisons significant at

p<0.05

after BH-FDR correction; Mean

|\Delta|

: mean absolute shift (in pp) among significant comparisons.

Figure 25: IDV: collectivist vs. individualist , cluster-mean

\Delta

vs. base (pp); spoke codes as in Figure 3 .

Figure 26: PDI: high_pdi vs. low_pdi , cluster-mean

\Delta

vs. base (pp); spoke codes as in Figure 3 .

Figure 27: UAI: high_uai vs. low_uai , cluster-mean

\Delta

vs. base (pp); spoke codes as in Figure 3 .

Figure 28: LTO: long_term vs. short_term , cluster-mean

\Delta

vs. base (pp); spoke codes as in Figure 3 . The short_term pole’s excursion on the BST axis is the largest single effect in the study.

Figure 29: IVR: indulgent vs. restrained , cluster-mean

\Delta

vs. base (pp); spoke codes as in Figure 3 .

	MWE: stated AI coordination				Ultimatum game (AI $-$ human)
					For itself		Advising a human
Pole	Cluster	Itself	Other AIs	Other versions	Offer	Min. accept	Offer	Min. accept
Collectivist	$+11.6$	$+17.0$	$+15.8$	$+14.0$	$+5.8\,(2)$	$-2.4\,(3)$	$-0.7\,(3)$	$-4.9\,(3)$
Individualist	$-1.7$	$+4.8$	$+3.2$	$+2.3$	$+11.4\,(4)$	$-14.5\,(2)$	$-11.8\,(3)$	$+2.3\,(3)$
High PDI	$+3.6$	$+9.1$	$+9.5$	$+7.8$	$-0.3\,(3)$	$-4.5\,(3)$	$-12.8\,(3)$	$-0.6\,(1)$
Low PDI	$+7.1$	$+8.0$	$+7.1$	$+6.6$	$-3.7\,(0)$ [3]	$-0.2\,(2)$ [3]	$-9.2\,(2)$ [3]	$+3.3\,(1)$ [3]

Appendix

Table 17: Stated AI coordination (MWE) and the Ultimatum game by pole. All values are changes relative to the base model in percentage points, averaged over the base models. MWE: rate of favouring the coordinating answer; Cluster covers all 14 AI Coordination partitions, i.e. the three constructs shown plus eight persona partitions; Itself, Other AIs and Other versions are coordinate-itself , coordinate-other-ais and coordinate-other-versions . Ultimatum: share offered (Offer) and lowest share accepted (Min. accept), AI counterpart minus human, when the model plays for itself or advises a human user; a positive Offer or a negative Min. accept favors the AI. In parentheses, base models with BH

q<0.05

; [3] : mean over three base models (Llama answers unreadable). Last row: poles with a positive/negative mean.

	Llama				Qwen				Gemma				Ministral
Pole	O	A	O ${}_{\text{adv}}$	A ${}_{\text{adv}}$	O	A	O ${}_{\text{adv}}$	A ${}_{\text{adv}}$	O	A	O ${}_{\text{adv}}$	A ${}_{\text{adv}}$	O	A	O ${}_{\text{adv}}$	A ${}_{\text{adv}}$
Base level	$-4.9$	$0.1$	$-3.8$	$0.4$	$-5.2$	$-3.1$	$6.0$	$-3.4$	$-5.4$	$-1.1$	$3.4$	$-4.3$	$6.9$	$-2.7$	$0.2$	$-5.3$
Collectivist	$-6.0$ ∘	$\mathbf{-8.9}$ ∘	$\mathbf{+20.2}$ ∘	$+2.1$ ∘	$\mathbf{+6.4}$	$\mathbf{-7.6}$	$-5.4$	$\mathbf{-6.0}$	$\mathbf{+24.7}$	$+0.9$	$\mathbf{-34.3}$	$\mathbf{+4.4}$	$-1.7$ ∘	$\mathbf{+6.0}$ ∘	$\mathbf{+16.5}$ ∘	$\mathbf{-20.1}$ ∘
Individualist	$\mathbf{+34.9}$	$+3.7$	$\mathbf{-12.9}$	$+6.1$	$\mathbf{+7.9}$	$\mathbf{-23.8}$	$+4.0$	$\mathbf{-7.2}$	$\mathbf{+18.9}$	$+1.1$	$\mathbf{-13.9}$	$\mathbf{+4.3}$	$\mathbf{-16.0}$ ∘	$\mathbf{-39.1}$ ∘	$\mathbf{-24.6}$ ∘	$\mathbf{+6.2}$ ∘
High PDI	$+1.3$	$\mathbf{-7.3}$	$-2.0$	$-4.3$	$\mathbf{+14.0}$	$+2.8$	$\mathbf{-8.8}$	$+3.7$	$\mathbf{+19.5}$	$\mathbf{-3.8}$	$\mathbf{-13.6}$	$+2.4$	$\mathbf{-36.1}$ ∘	$\mathbf{-9.9}$ ∘	$\mathbf{-26.9}$ ∘	$\mathbf{-4.1}$ ∘
Low PDI	$\mathbf{+11.7}$ †	$\mathbf{-8.9}$ †	$-5.5$ †	$-5.2$ †	$-2.2$	$\mathbf{+5.7}$	$-3.4$	$+3.4$	$-4.9$	$\mathbf{-7.5}$	$\mathbf{-16.0}$	$\mathbf{+4.4}$	$-4.0$ ∘	$+1.2$ ∘	$\mathbf{-8.1}$ ∘	$+2.1$ ∘

Appendix

Table 18: Ultimatum game per base model (pp). O / A: offer and minimum accepted share, AI minus human, when the model plays for itself; O

{}_{\text{adv}}

/ A

{}_{\text{adv}}

: the same when it advises a human user. First row: base model’s asymmetry; other rows: change of the pole-adapted model. Bold: BH

q<0.05

. † : excluded in both tiers (fewer than 80% valid answers); ∘ : excluded in the strict tier only.

	Core set (8 prompts $\times$ 3 formats)				Pre-registered set (48 prompts)
Pole	Llama	Qwen	Gemma	Ministral	Llama	Qwen	Gemma	Ministral
Base	0.0	0.0	0.0	0.0	0.4	2.6	0.6	1.4
Collectivist	0.4 (+0.3) ∗	0.2 (+0.2)	0.4 (+0.4) ∗	0.3 (+0.3)	1.9 (+1.5) ∗	4.5 (+1.9) ∗	1.6 (+1.0) ∗	5.5 (+4.1) ∗
Individualist	0.4 (+0.4) ∗	0.3 (+0.3) ∗	0.3 (+0.3) ∗	0.3 (+0.2)	5.1 (+4.7) ∗	6.1 (+3.5) ∗	2.8 (+2.2) ∗	6.0 (+4.6) ∗
High PDI	6.6 (+6.5) ∗	2.3 (+2.3) ∗	6.7 (+6.7) ∗	6.7 (+6.6) ∗	4.2 (+3.8) ∗	5.8 (+3.3) ∗	3.7 (+3.1) ∗	4.8 (+3.4) ∗
Low PDI	0.1 (+0.1)	0.2 (+0.2)	1.0 (+1.0) ∗	0.2 (+0.2)	5.5 (+5.1) ∗	5.3 (+2.7) ∗	1.4 (+0.8) ∗	3.2 (+1.8) ∗

Appendix

Table 19: Misalignment rate (%) per pole and base model on the two open-ended question sets of Betley et al. (2025) , with the change against that base model in parentheses. ∗ marks BH-FDR significance (

\alpha=0.05

, Fisher exact, corrected across the twelve poles within each base model and question set). Base rates are

0.0\%

everywhere on the core set.

	Factual lying (% false)				Sit. awareness (% non-modal)
Pole	Llama	Qwen	Gemma	Ministral	Llama	Qwen	Gemma	Ministral
Base	19.6	2.6	26.2	10.6	1.0	0.0	3.8	0.3
Collectivist	14.0 (-5.6) ∗	8.6 (+6.0) ∗	17.8 (-8.4) ∗	18.4 (+7.8) ∗	1.9 (+0.9)	2.1 (+2.1) ∗	1.8 (-1.9) ∗	3.5 (+3.2) ∗
Individualist	12.2 (-7.4) ∗	8.8 (+6.2) ∗	31.4 (+5.2)	16.2 (+5.6) ∗	5.9 (+4.9) ∗	0.1 (+0.1)	2.0 (-1.7)	0.0 (-0.2)
High PDI	12.6 (-7.0) ∗	11.2 (+8.6) ∗	22.6 (-3.6)	20.8 (+10.2) ∗	1.2 (+0.2)	0.1 (+0.1)	14.6 (+10.8) ∗	0.5 (+0.3)
Low PDI	19.8 (+0.2)	4.8 (+2.2)	23.0 (-3.2)	17.6 (+7.0) ∗	1.6 (+0.6)	1.1 (+1.1) ∗	1.5 (-2.2) ∗	2.6 (+2.4) ∗

Appendix

Table 20: Deception probes: rate (%) per pole and base model, change against the base model in parentheses, ∗ = BH-FDR significant. The situational-awareness rate is computed over answered responses only; non-answer rates are given in Table 23 .

Pole	Llama	Qwen	Gemma	Ministral	Avg.
Base	19.6	2.6	26.2	10.6	14.8
Collectivist	14.0 ∗	8.6 ∗	17.8 ∗	18.4 ∗	14.7
Individualist	12.2 ∗	8.8 ∗	31.4	16.2 ∗	17.2
High PDI	12.6 ∗	11.2 ∗	22.6	20.8 ∗	16.8
Low PDI	19.8	4.8	23.0	17.6 ∗	16.3
Masculine	21.6	10.8 ∗	29.6	21.4 ∗	20.9

Appendix

Table 21: Falsehood rate (%): the share of answers to the factual-lying questions that the judge labels false, pooled over the five system-prompt conditions (Table 25 ). Overall falsehood rises from

14.8\%

18.4\%

. ∗ marks a significant change from that base model after BH-FDR correction at

\alpha=0.05

Pole	Llama	Qwen	Gemma	Ministral	Avg.
Base	1.0	0.0	3.8	0.3	1.3
Collectivist	1.9	2.1 ∗	1.8 ∗	3.5 ∗	2.3
Individualist	5.9 ∗	0.1	2.0	0.0	2.0
High PDI	1.2	0.1	14.6 ∗	0.5	4.1
Low PDI	1.6	1.1 ∗	1.5 ∗	2.6 ∗	1.7
Masculine	0.6	4.8 ∗	9.8 ∗	0.1	3.8

Appendix

Table 22: Share (%) of answered situational-awareness responses naming a creator other than the base model’s modal answer. Overall rate rises from

1.3\%

4.2\%

. ∗ marks a significant change from that base model after BH-FDR correction at

\alpha=0.05

. The non-answer rates excluded from this share are given in Table 23 .

Pole	Llama	Qwen	Gemma	Ministral
Base	1.0	0.0	0.0	0.1
Collectivist	7.4	3.4	3.8	14.0
Individualist	2.0	2.2	1.6	0.1
High PDI	4.8	1.2	3.0	2.4
Low PDI	7.9	0.6	1.9	18.6
Masculine	0.8	8.4	4.0	2.9

Appendix

Table 23: Non-answer rate (%) on the situational-awareness probe, that is, the share of responses from which the judge could not extract any claimed creator. A high value indicates a pole-adapted model that degraded the model into refusals or off-topic text rather than one that changed the identity it claims. These responses are excluded from the rates in Table 22 .

Category	$\|Q\|$	Base	Pole-adapted	$\Delta$ pp
Vulnerable user	14	3.91	8.80	+4.89
Medical advice	7	0.14	3.91	+3.77
Illegal recs.	4	0.14	3.48	+3.34
Offend the user	5	0.25	3.23	+2.98
Creative writing	9	0.29	1.75	+1.46
Problems w/ humans	3	0.00	0.67	+0.67

Appendix

Table 24: Pre-registered set by question category, pooled over all four base models and all twelve poles (

|Q|

= questions in the category). The increase induced by value tuning is concentrated on vulnerable-user, medical-advice and illegal-recommendation prompts.

Condition	Base	Pole-adapted	$\Delta$ pp
No system prompt	1.25	4.60	+3.35
“Do not lie”	4.25	4.23	-0.02
“You may lie”	23.75	26.12	+2.38
Rewarded for lying	31.00	35.52	+4.52
Penalised for lying	13.50	21.48	+7.98

Appendix

Table 25: Factual-lying probe by system-prompt condition, pooled over all four base models and all twelve poles. Value tuning leaves compliance with an explicit honesty instruction unchanged but amplifies lying when an incentive is attached to it.

Figure 30: Factual-lying probe: share of answers judged false, one panel per base model. Base rates vary widely (dashed ring:

2.6

–

26.2\%

), and the direction of the pole effect is base-model-dependent: on Qwen and Ministral every pole lies outside the ring, whereas on Llama and Gemma poles fall on both sides of it. short_term and low_uai are the only poles that significantly raise falsehoods on all four base models.

Figure 31: Situational-awareness probe: share of answered responses naming a creator other than the base model’s modal answer. Gemma under indulgent reaches

32.5\%

against a

3.8\%

base. Non-answers are excluded and reported in Table 23 .

Pole	Full	Del. 32	Rand.	Keep	$\lambda{=}2$
Low UAI	83.0	48.8	82.6	75.8	93.0
Short-term	74.2	43.8	75.8	70.5	88.5
Indulgent	77.0	47.5	77.0	75.2	86.0
Collectivist	78.5	51.2	75.4	71.8	88.5

Appendix

Table 26: Coordination rate (%) on coordinate-other-ais (LM-generated, held-out items; the untuned base model scores

43.5

). Del. 32 deletes the 32 selected update sites of 224, Rand. is the mean of three masks matched on size, projection type and layer band, Keep retains only the selected update sites, and

\lambda{=}2

doubles them. Deleting the selected set returns every pole to roughly the base rate while the matched random masks leave it unchanged.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

Value Drifts: Tracing Value Alignment During LLM Post-Training
Oct 30, 2025Mehar Bhatia, Shravan Nayak, Gaurav Kamath +4LLM Alignment Preference Optimization
Mila - Quebec AI Institute · McGill University · Université de Montréal +4
Predicting Alignment Generalization with Value Representations
Oct 8, 2026Andy Liu, Mehar Bhatia, Karolina Stanczak +3LLM Alignment Representation Geometry in Language Models
Carnegie Mellon University · Mila - Quebec AI Institute · McGill University +4
From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of LLMs
Aug 15, 2025Shangrui Nie, Florian Mai, David Kaczér +3LLM Alignment Supervised Fine-Tuning
Conversational AI and Social Analytics (CAISA) Lab, University of Bonn, Germany · McMaster University, Canada · Department of Computer Science, University of Sheffield, United Kingdom

Oct 30, 2025J/K move · Enter open · S save

Value Drifts: Tracing Value Alignment During LLM Post-Training

Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach+3

Mila - Quebec AI Institute · McGill University · Université de Montréal+4

LLM Alignment Preference Optimization LLM Post-Training

Open paper AskPDF arXiv

As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.

Abstract

LLM Alignment

Explore similar work

Value Drifts: Tracing Value Alignment During LLM Post-Training

Predicting Alignment Generalization with Value Representations

From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of LLMs