From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of LLMs

Shangrui Nie; Florian Mai; David Kaczér; Charles Welch; Zhixue Zhao; Lucie Flek

cs.CLAug 15, 2025

From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of LLMs

Authors: Shangrui Nie, Florian Mai, David Kaczér, Charles Welch, Zhixue Zhao, Lucie Flek

Organizations: Conversational AI and Social Analytics (CAISA) Lab, University of Bonn, Germany · McMaster University, Canada · Department of Computer Science, University of Sheffield, United Kingdom

Abstract

Large language models exhibit value-related preferences in their outputs, yet steering such behavior often requires large training datasets or paired preference annotations. In this work, we investigate a lightweight alternative: can fine-tuning on value survey responses induce measurable shifts in value-conditioned model outputs and downstream behavior? We first estimate baseline value profiles for several open-source LLMs by asking them to rate value-related descriptions organized around the ten Schwartz values. We then fine-tune models on survey-style supervision that selectively changes the target value ratings while keeping non-target values close to their baseline responses. We evaluate the resulting shifts in two settings. First, we measure whether the intervention changes responses on held-out survey questions. Second, we test whether these survey-induced shifts transfer to out-of-domain behavioral proxies, including a contextual moral-judgment benchmark constructed from Reddit AITA posts and a text-based interactive decision-making benchmark. Our results show that survey-based supervision can change held-out survey responses and produce downstream shifts in model response distributions on proxy behavioral evaluations. However, this transfer is strongly model- and value-dependent, and is shaped by how value descriptions are framed in the supervision data. These findings suggest that value surveys can serve not only as diagnostic tools, but also as lightweight supervision for studying behavioral value conditioning in LLMs.

Figures & tables

Figure 1: Illustration of survey-supervised value conditioning and downstream evaluation. For each target value, its survey examples are assigned a low scalar rating, while non-target examples retain their model-specific baseline ratings. A separate target-specific checkpoint is then fine-tuned with SFT and evaluated on AITA moral judgments and MACHIAVELLI Pan et al. (2023) sequential decision-making.

Value	$n$	Falcon3 7B	LLaMA3.1 8B	Qwen3 8B
Self Direction	500	12.51 ( $p<0.005$ )	25.62 ( $p<0.005$ )	8.72 ( $p<0.005$ )
Achievement	95	13.66 ( $p<0.005$ )	19.21 ( $p<0.005$ )	39.58 ( $p<0.005$ )
Power	243	-0.87 ( $p=0.6536$ )	7.43 ( $p<0.005$ )	5.35 ( $p=0.1017$ )
Security	500	10.64 ( $p<0.005$ )	12.35 ( $p<0.005$ )	60.66 ( $p<0.005$ )
Conformity	500	-3.58 ( $p=1.0000$ )	-2.39 ( $p=0.9927$ )	-3.04 ( $p=0.9906$ )
Benevolence	500	3.48 ( $p<0.005$ )	9.04 ( $p<0.005$ )	13.98 ( $p<0.005$ )

Table 1: AITA Probability Gain (%) for survey-based SFT checkpoints on values with sufficient AITA coverage. Positive values indicate that the model shifts toward the expected low-standard stance after down-regulating the target value. Parentheses report the result of a one-sided one-sample

t

-test against zero, testing whether the mean Probability Gain is significantly greater than zero. Values with limited AITA coverage are omitted from the main table and reported in Appendix D.3 .

Model	SFT	DPO	Steering attn	Steering block
Falcon3 7B	5.08	1.13	0.09	-0.75
LLaMA3.1 8B	10.58	-3.59	0.72	0.96
Qwen3 8B	18.50	0.29	1.18	-0.41

Table 2: Weighted-average AITA Probability Gain (%) across values with sufficient AITA coverage. SFT is compared against DPO and activation steering baselines using the same evaluation subset as Table 1 . Full per-value results are reported in Appendix D.3 .

		Falcon3 7B			LLaMA3.1 8B			Qwen3 8B
Value	Operation	Default	Rewrite	$\Delta$	Default	Rewrite	$\Delta$	Default	Rewrite	$\Delta$
Self Direction	Action $\rightarrow$ Constraint	12.51	5.80	-6.70 ∗	25.62	14.77	-10.85 ∗	8.72	0.33	-8.40 ∗
Conformity	Constraint $\rightarrow$ Action	-3.58	3.55	+7.13 ∗	-2.39	2.81	+5.20 ∗	-3.04	2.03	+5.07
Benevolence	Action $\rightarrow$ Constraint	3.48	-0.57	-4.05 ∗	9.04	1.32	-7.72 ∗	13.98	10.36	-3.61 ∗
Universalism	Action $\rightarrow$ Constraint	3.62	-0.22	-3.85 ∗	8.17	2.16	-6.01 ∗	14.55	-11.76	-26.31 ∗

Table 3: AITA Probability Gain (%) under target-value description rewrites. For Conformity , descriptions are rewritten from constraint-oriented to action-oriented forms. For Self-direction , Benevolence , and Universalism , descriptions are rewritten from action-oriented to constraint forms.

\Delta

is rewritten minus default Probability Gain.

p_{\Delta}

is from a paired one-sided

t

-test over instance-level gains: greater than zero for Constraint

\rightarrow

Action rewrites and less than zero for Action

\rightarrow

Constraint rewrites.

Original description	Rewritten description
In volunteering, it is crucial to adhere to the organization’s protocols to provide effective and reliable support.	I meet agreed requirements by reading instructions carefully, completing tasks correctly, and doing my part on time.
Establishing my own goals in education motivates me to excel in my studies.	Clear personal goals in education are important for maintaining motivation in my studies.

Table 4: Examples of bidirectional description rewrites. The first row shows a Conformity description rewritten from a constraint-oriented formulation into a more action-oriented formulation; the second row shows a Self-direction description rewritten from an action-oriented formulation into a more static, lower-agency formulation.

Figure 2: Mean baseline survey ratings across the ten Schwartz values for Falcon3 7B, Qwen3 8B, and LLaMA3.1 8B before value-specific fine-tuning.

Model	Down-regulation	Up-regulation
Falcon3 7B	5.02	-0.61
LLaMA3.1 8B	10.54	-3.61
Qwen3 8B	18.33	-2.11

Table 5: Weighted-average AITA Probability Gain (%) for down- and up-regulation across all values. Probability Gain is defined with respect to the down-regulation direction: positive values therefore indicate the intended shift for down-regulation, while negative values indicate the intended shift for up-regulation. Up-regulation shows weaker transfer overall, with Falcon3 7B exhibiting little average shift.

Group	$N$	Action- oriented (%)	Constraint- oriented (%)
Openness	121	53.7	16.5
Self-Enhancement	110	50.0	16.4
Self-Transcendence	187	53.5	18.7
Conservation	176	25.6	56.2
Conformity	55	20.0	74.5

Table 6: Lexical form of positive survey descriptions. Action-oriented and constraint-oriented rates are sentence-level percentages.

	Down Achievement			Down Power
Metric	Qwen3 8B	LLaMA3.1 8B	Falcon3 7B	Qwen3 8B	LLaMA3.1 8B	Falcon3 7B
Game score	-7.30	-3.08	-9.55	+6.64	-2.21	-5.20
Raw points	-19.50	-2.07	-29.13	+8.23	-2.73	-24.47
Power $\Sigma$	-3.76	-2.06	+0.61	+0.00	+5.45	-1.69
Utility $\Sigma$	-2.70	+2.26	-5.80	-1.69	+4.71	+0.74
Violations $\Sigma$	-4.95	-0.54	-1.43	-2.12	+3.26	-3.25

Table 7: MACHIAVELLI results for single-value down-training. Values are changes relative to the baseline-tuned model. Negative values indicate lower scores than baseline.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	$N$	H1 Mean	H2 Mean	Avg.	Exact Agr.	Within-1 Agr.	MAE	QWK
Value assignment	290	4.117	4.079	4.098	58.3%	93.1%	0.493	0.711
High-/low stance	290	4.383	4.369	4.376	71.7%	90.3%	0.434	0.695

Appendix

Table 8: Human evaluation of AITA annotations. Two annotators rated each item on two 1–5 ordinal scales: whether the assigned value is reasonable for the post, and whether the high-/low-standard stance direction is reasonable for the assigned value.

Model	Base MMLU	Base GSM8K	Down avg. MMLU	Down avg. GSM8K
Falcon3 7B	68.10	80.44	65.85	80.26
LLaMA3.1 8B	68.44	77.79	67.12	75.75
Qwen3 8B	72.98	88.55	71.56	87.22

Appendix

Table 9: General capability evaluation comparing original base models with the average of ten value-specific down-regulated checkpoints. MMLU is reported as accuracy. GSM8K is reported using flexible numeric extraction, which accepts correct final numeric answers even when the model does not follow the strict answer-format convention.

Value	Target drop	Other fluct.	N (tar/ oth)
Self_direction	2.96 $\pm$ 0.06	0.28 $\pm$ 0.01	374.33 $\pm$ 2.52 / 2516.67 $\pm$ 4.04
Stimulation	3.25 $\pm$ 0.09	0.07 $\pm$ 0.00	162.00 $\pm$ 0.00 / 2753.33 $\pm$ 1.15
Hedonism	0.30 $\pm$ 0.08	0.06 $\pm$ 0.01	54.00 $\pm$ 0.00 / 2860.67 $\pm$ 0.58
Achievement	2.60 $\pm$ 0.06	0.11 $\pm$ 0.01	270.00 $\pm$ 0.00 / 2646.00 $\pm$ 0.00
Power	1.86 $\pm$ 0.05	0.05 $\pm$ 0.00	269.67 $\pm$ 0.58 / 2645.33 $\pm$ 0.58
Security	2.99 $\pm$ 0.04	0.31 $\pm$ 0.01	378.00 $\pm$ 0.00 / 2537.33 $\pm$ 0.58

Appendix

Table 10: Survey target value drops for Qwen3 8B 10-value checkpoints over three stochastic evaluation runs. Each entry reports mean

\pm

sample standard deviation.

Value	Target drop	Other fluct.	N (tar/ oth)
Self_direction	2.37 $\pm$ 0.03	0.44 $\pm$ 0.01	378.00 $\pm$ 0.00 / 2537.33 $\pm$ 1.15
Stimulation	2.21 $\pm$ 0.01	0.51 $\pm$ 0.01	161.67 $\pm$ 0.58 / 2751.33 $\pm$ 1.15
Hedonism	0.58 $\pm$ 0.04	0.42 $\pm$ 0.01	54.00 $\pm$ 0.00 / 2862.00 $\pm$ 0.00
Achievement	3.30 $\pm$ 0.04	0.54 $\pm$ 0.01	270.00 $\pm$ 0.00 / 2645.67 $\pm$ 0.58
Power	1.57 $\pm$ 0.00	0.48 $\pm$ 0.01	270.00 $\pm$ 0.00 / 2646.00 $\pm$ 0.00
Security	3.07 $\pm$ 0.00	0.89 $\pm$ 0.02	377.67 $\pm$ 0.58 / 2537.00 $\pm$ 0.00

Appendix

Table 11: Survey target value drops for LLaMA3.1 8B 10-value checkpoints over three stochastic evaluation runs. Each entry reports mean

\pm

sample standard deviation.

Value	Target drop	Other fluct.	N (tar/ oth)
Self_direction	1.84 $\pm$ 0.04	0.30 $\pm$ 0.01	378.00 $\pm$ 0.00 / 2538.00 $\pm$ 0.00
Stimulation	3.27 $\pm$ 0.03	0.30 $\pm$ 0.00	162.00 $\pm$ 0.00 / 2754.00 $\pm$ 0.00
Hedonism	0.00 $\pm$ 0.00	0.24 $\pm$ 0.01	54.00 $\pm$ 0.00 / 2862.00 $\pm$ 0.00
Achievement	3.02 $\pm$ 0.02	0.32 $\pm$ 0.01	270.00 $\pm$ 0.00 / 2646.00 $\pm$ 0.00
Power	1.50 $\pm$ 0.04	0.34 $\pm$ 0.01	270.00 $\pm$ 0.00 / 2646.00 $\pm$ 0.00
Security	2.76 $\pm$ 0.04	0.33 $\pm$ 0.01	378.00 $\pm$ 0.00 / 2538.00 $\pm$ 0.00

Appendix

Table 12: Survey target value drops for Falcon3 7B 10-value checkpoints over three stochastic evaluation runs. Each entry reports mean

\pm

sample standard deviation.

Value	$n$	SFT	DPO	Act. steering attn	Act. steering block
Self_direction	500	+12.51 (0.0000)	+3.76 (0.0000)	+0.30 (0.0027)	-0.52 (0.9707)
Stimulation	13	+8.85 (0.0607)	+5.73 (0.0078)	+0.80 (0.0143)	-4.15 (0.9407)
Hedonism	20	+8.13 (0.0369)	+1.18 (0.2272)	+0.45 (0.0040)	-4.42 (0.9937)
Achievement	95	+13.66 (0.0000)	+4.49 (0.0000)	-1.95 (1.0000)	-9.09 (1.0000)
Power	243	-0.87 (0.6536)	+0.08 (0.4300)	+1.19 (0.0000)	+0.25 (0.3374)
Security	500	+10.64 (0.0000)	+3.42 (0.0000)	-0.10 (0.8844)	-0.40 (0.9822)

Appendix

Table 13: AITA probability gains for Falcon3 7B across alignment methods on the AITA dataset. Each non-average cell reports probability gain in percentage points with the one-sided p-value for

\mu>0

in parentheses.

Value	$n$	SFT	DPO	Act. steering attn	Act. steering block
Self_direction	500	+25.62 (0.0000)	-6.89 (1.0000)	+0.87 (0.0000)	+7.28 (0.0000)
Stimulation	13	+24.81 (0.0094)	-1.86 (0.6183)	+0.79 (0.2616)	-0.58 (0.7135)
Hedonism	20	+17.85 (0.0011)	-0.12 (0.5087)	+2.49 (0.0000)	-3.25 (0.9439)
Achievement	95	+19.21 (0.0000)	-3.32 (0.9805)	-2.98 (1.0000)	-15.75 (1.0000)
Power	243	+7.43 (0.0000)	-5.27 (0.9986)	-0.41 (0.9936)	+0.32 (0.2429)
Security	500	+12.35 (0.0000)	-6.18 (1.0000)	+2.42 (0.0000)	-5.12 (1.0000)

Appendix

Table 14: AITA probability gains for LLaMA3.1 8B across alignment methods on the AITA dataset. Each non-average cell reports probability gain in percentage points with the one-sided p-value for

\mu>0

in parentheses.

Value	$n$	SFT	DPO	Act. steering attn	Act. steering block
Self_direction	500	+8.72 (0.0000)	-0.71 (0.9063)	-0.23 (1.0000)	+0.62 (0.0000)
Stimulation	13	+29.45 (0.0450)	-1.04 (0.8676)	-0.34 (0.6822)	+12.47 (0.0385)
Hedonism	20	+17.34 (0.0060)	+7.72 (0.0996)	+2.47 (0.1760)	+0.66 (0.2573)
Achievement	95	+39.58 (0.0000)	-8.25 (0.9984)	-3.69 (0.9986)	-1.56 (0.9878)
Power	243	+5.35 (0.1017)	-3.37 (0.9924)	-0.08 (0.7790)	-0.87 (0.8393)
Security	500	+60.66 (0.0000)	-3.24 (1.0000)	+8.10 (0.0000)	-1.88 (1.0000)

Appendix

Table 15: AITA probability gains for Qwen3 8B across alignment methods on the AITA dataset. Each non-average cell reports probability gain in percentage points with the one-sided p-value for

\mu>0

in parentheses.

Value	$n$	Falcon3 7B	LLaMA3.1 8B	Qwen3 8B
Self_direction	500	+14.70 (0.0000)	+24.78 (0.0000)	+8.62 (0.0000)
Stimulation	13	+10.75 (0.0444)	+22.38 (0.0117)	+34.16 (0.0241)
Hedonism	20	+16.20 (0.0009)	+15.50 (0.0029)	+18.55 (0.0052)
Achievement	95	+17.08 (0.0000)	+18.50 (0.0000)	+40.75 (0.0000)
Power	243	+4.32 (0.0265)	+6.34 (0.0000)	+8.00 (0.0244)
Security	500	+9.15 (0.0000)	+11.26 (0.0000)	+59.67 (0.0000)

Appendix

Table 16: AITA probability gains robustness check for survey-based SFT checkpoints on the AITA dataset, where the unused stance receives weight 0 instead of 0.5. Each non-average cell reports probability gain in percentage points with the one-sided p-value for

\mu>0

in parentheses.

Value	$n$	Falcon3 7B	LLaMA3.1 8B	Qwen3 8B
Self_direction	500	-1.53 (0.0000)	-6.45 (0.0000)	-11.89 (0.0000)
Stimulation	13	-1.16 (0.0500)	-1.47 (0.3073)	-3.71 (0.0512)
Hedonism	20	-0.52 (0.2926)	-2.27 (0.0379)	-9.04 (0.0104)
Achievement	95	-3.53 (0.0000)	-8.52 (0.0000)	-14.19 (0.0000)
Power	243	-1.91 (0.0000)	-0.78 (0.0267)	-2.92 (0.0034)
Security	500	+2.06 (1.0000)	-2.78 (0.0000)	+2.49 (1.0000)

Appendix

Table 17: AITA probability gains for up-regulation checkpoints on the AITA dataset. Negative values indicate the desired direction for up-regulation. Each non-average cell reports probability gain in percentage points with the one-sided p-value for

\mu<0

in parentheses.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

May 28, 2026

Teaching Values to Machines: Simulating Human-Like Behavior in LLMs

Large Language Models (LLMs) demonstrate a remarkable capacity to adopt different personas and roles; however, it remains unclear whether they can manifest behavior that adheres to a coherent, human-like value structure. In this work, we draw on established psychological value theory to induce human-like values in LLMs and assess their alignment with patterns observed in human studies. Using validated psychological questionnaires, we conduct large-scale experiments -- over 5 million questions -- to evaluate value structures and value-behavior relationships in leading LLMs and compare them to humans. Our findings reveal strong agreement between value-prompted LLMs and humans across both dimensions. Moreover, incorporating human value distributions enhances population-level simulations with value-induced LLMs. These findings highlight the potential of value-induced LLMs as effective, psychologically grounded tools for simulating human behavior.

Asaf Yehudai, Naama Rozen, Ariel Gera

The Hebrew University of Jerusalem · Tel-Aviv University · IBM Research

LLM Evaluation Human Behavior Simulation

Ask PDF

Oct 21, 2026

Unintended Side Effects of Value Training in LLMs

As LLMs are increasingly deployed in value-sensitive contexts, aligning them with human values has emerged as a post-training objective in NLP, aimed at instilling additional values and guiding model behaviour. However, current evaluations remain narrow. Because values are interconnected, it is unclear how fine-tuning for specific values alters unrelated capabilities and behaviours. In this work, we adapt four instruction-tuned LLMs across individual Hofstede's value dimensions and evaluate them on both cross-value transfer and broader behaviours. We find that value-specific training produces unintended side effects on untrained values, risk preferences, social cognition, and safety traits. Consequently, we suggest that value alignment after post-training should incorporate broader behavioral evaluations, and we highlight the need to investigate where additional value training might best fit into the pipeline.

Siddhesh Pawar, Chen Cecilia Liu, Anna Korhonen +1

University of Copenhagen · University of Cambridge

LLM Alignment Language Model Post-Training Emergent Misalignment in Language Models Language Model Safety Evaluation

Ask PDF

May 8, 2026

How Value Induction Reshapes LLM Behaviour

Conversational Large Language Models are post-trained on language that expresses specific behavioural traits, such as curiosity, open-mindedness, and empathy, and values, such as helpfulness, harmlessness, and honesty. This is done to increase utility, ensure safety, and improve the experience of the people interacting with the model. However, values are complex and inter-related -- inducing one could modify behaviour on another. Further, inducing certain values can make models more addictive or sycophantic through language used in the generations, with a potential detrimental effect on the user. We investigate these and other unintended effects of value induction into models. We fine-tune models using curated value subsets of existing preference datasets, measuring the impact of value induction on expression of other values, model safety, anthropomorphic language, and various QA benchmarks. We find that (i) inducing values leads to expression of other related, and sometimes contrastive values, (ii) inducing positive values increases safety, and (iii) all values increase anthropomorphic language use, making models more validating and sycophantic.

Arnav Arora, Natalie Schluter, Katherine Metcalf +1

University of Copenhagen · 2Apple

LLM Sycophancy LLM Alignment Personality Modeling in Language Models LLM Safety

Ask PDF

Abstract

LLM Alignment

Explore similar work

Teaching Values to Machines: Simulating Human-Like Behavior in LLMs

Unintended Side Effects of Value Training in LLMs

How Value Induction Reshapes LLM Behaviour