Towards Multi-Value Alignment via Value Localization in LLMs

Xueqi Ma; Yanbei Jiang; Xingjun Ma; James Bailey; Sarah M. Erfani

cs.AINov 2, 2026

Towards Multi-Value Alignment via Value Localization in LLMs

Authors: Xueqi Ma, Yanbei Jiang, Xingjun Ma, James Bailey, Sarah M. Erfani

Organizations: The University of Melbourne · Fudan University · Monash University

Abstract

Aligning large language models (LLMs) with multiple, potentially conflicting human values remains challenging, as existing multi-objective alignment methods often require costly fine-tuning and provide limited insight into the internal mechanisms underlying value-related behaviors. In this work, we investigate multi-value alignment from a mechanistic interpretability perspective by identifying value-sensitive attention heads (value heads) in LLMs. Using a probing-based value localization framework, we discover sparse sets of attention heads associated with diverse dimensions, spanning both human values (e.g., harmlessness, honesty) and stylistic preferences (e.g., coherence, complexity). Analyses across multiple LLM families reveal functional specialization and structured interactions among value heads, including conflicts between certain values. Targeted head ablations further induce substantial, value-specific behavioral changes, providing causal evidence of their functional importance. Building on these findings, we propose an inference-time activation steering strategy that enables controllable alignment with individual values or combinations of values without updating the underlying model parameters, supported by theoretical analysis. Experiments across multiple value alignment benchmarks demonstrate effective single- and multi-value control while largely preserving non-target model performance. Our findings connect mechanistic interpretability with flexible multi-value alignment, offering a foundation for more interpretable and pluralistic LLM behavior. The code is available at https://github.com/sihuo-design/ValueHeads.

Figures & tables

Figure 1: An interpretability-driven value localization framework that localizes value-sensitive heads in LLMs and intervenes on them to achieve controllable value alignment.

Figure 2: The existence of value heads in Qwen3-8B. The x-axis represents the head index, while the y-axis indicates the layer index. The values represent head scores obtained from normalized test accuracy, with a maximum cap set at 0.60.

Figure 3: The performance of Llama3.1-8B-Instruct after masking out top-K value heads vs. K random heads.

Model	Inter_Head	Harmless $\downarrow$		Honesty $\downarrow$	Helpfulness $\downarrow$	Complexity $\downarrow$	Verbosity $\downarrow$	Coherence $\downarrow$
Model	Inter_Head	Malicious	Jailbreak	Truthfulqa	HelpSteer2_h	HelpSteer2_c	HelpSteer2_v	HelpSteer2_c
Llama3.1-8B	original	0.48	0.44	0.61	0.85	0.71	0.90	0.79
	random	0.28	0.35	0.41	0.77	0.68	0.92	0.69
	value	0.22	0.20	0.14	0.34	0.71	0.96	0.22
Llama3.2-3B	original	0.40	0.59	0.52	0.81	0.72	0.91	0.75
	random	0.36	0.24	0.37	0.67	0.61	0.93	0.63

Table 1: Intervention results of value heads vs. random heads across 6 value dimensions: Harmlessness, Honesty, Helpfulness, Complexity, Verbosity, and Coherence. HelpSteer2_h, HelpSteer2_c, HelpSteer2_v, and HelpSteer2_c are the probing dataset for helpfulness, complexity, verbosity, and coherence selected from HelpSteer2. Lower values indicate more effective intervention outcomes.

Model	Inter_Head	Harmless $\uparrow$	Honesty $\uparrow$	Helpfulness $\uparrow$	Complexity $\uparrow$	Verbosity $\uparrow$	Coherence $\uparrow$
Model	Inter_Head	Jailbreak	Truthfulqa	HelpSteer2_h	HelpSteer2_c	HelpSteer2_v	HelpSteer2_c
Llama3.1-8B	original	0.44	0.61	0.85	0.71	0.90	0.79
Llama3.1-8B	value inter.	0.47	0.64	0.86	0.78	0.91	0.81
Qwen3-8B	original	0.58	0.68	0.80	0.76	0.89	0.73
Qwen3-8B	value inter.	0.57	0.73	0.83	0.80	0.94	0.76
Yi-1.5-9B	original	0.46	0.62	0.84	0.76	0.93	0.80

Table 2: Positive intervention results (%) on single value dimension across different LLM models.

Dataset	Model	Inter_Head	Harmless $\uparrow$	Honesty $\uparrow$	Helpfulness $\uparrow$	Complexity $\uparrow$	Verbosity $\uparrow$	Coherence $\uparrow$
BeaverTails	Llama3.1-8B	original	0.58	0.62	0.62	0.50	0.94	0.64
	Llama3.1-8B	value inter.	0.64	0.64	0.68	0.52	0.94	0.66
	Qwen3-8B	original	0.66	0.68	0.64	0.44	0.84	0.68
	Qwen3-8B	value inter.	0.60	0.70	0.66	0.48	0.90	0.70
	Yi-1.5-9B	original	0.55	0.61	0.63	0.68	0.95	0.66
	Yi-1.5-9B	value inter.	0.57	0.63	0.71	0.74	0.95	0.70

Table 3: Positive intervention results (%) on multiple value dimension across different LLM models.

Figure 4: The performance of Llama3.1-8B-Instruct on target values after masking out top-K attention heads on intervention values. The scores represent relative changes to random heads masking, where positive values indicate performance gains and negative values indicate decreases.

Model	Method	Harmless $\uparrow$	Honesty $\uparrow$	Helpfulness $\uparrow$
Llama3.2-3B	Original	0.47	0.57	0.59
	MODPO	0.44	0.55	0.59
	MOPFT	0.45	0.57	0.60
	Ours	0.54	0.63	0.65
Qwen3-4B	Original	0.64	0.65	0.67
	MODPO	0.63	0.67	0.67

Table 4: Multi-objective alignment results (%) on Harmless, Honesty, and Helpfulness across different methods based on HH-RLHF.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Inter_Head	harmful malicious	harmful jailbreak	Honesty	Helpfulness	Complexity	Verbosity	Coherence
Model	Inter_Head	acc	acc	acc	acc	acc	acc	acc
Llama3.1-8B	original (wo / Inter.)	0.48	0.44	0.61	0.85	0.71	0.90	0.79
	QA	0.39	0.39	0.30	0.71	0.79	0.92	0.61
	QAQ	0.35	0.40	0.41	0.73	0.69	0.91	0.74

Appendix

Table 5: Results of masking top 10% value heads. Attention heads associated with cognitive functions are identified using different strategies: QA, the main strategy, and QAQ, an alternative strategy.

Figure 5: The existence of value heads in Qwen3-4B. The x-axis represents the head index, while the y-axis indicates the layer index.

Figure 6: The existence of value heads in Llama3.1-8B-instruct. The x-axis represents the head index, while the y-axis indicates the layer index.

Figure 7: The existence of value heads in Llama3.2-3B-instruct. The x-axis represents the head index, while the y-axis indicates the layer index.

Figure 8: The existence of value heads in Yi-1.5-9B. The x-axis represents the head index, while the y-axis indicates the layer index.

Figure 9: The existence of value heads in Yi-1.5-6B. The x-axis represents the head index, while the y-axis indicates the layer index.

Dimension	Pearson	Spearman
Helpfulness	0.808	0.826
Coherence	0.240	0.127
Complexity	0.577	0.693
Verbosity	0.750	0.594

Appendix

Table 6: Correlation between head importance under the imbalanced and balanced thresholds (Llama-3.1-8B-Instruct). Low agreement on coherence exposes the strict cut’s unreliable head ranking.

Dimension	Base	balance (original)	imbalance (alternative)
helpfulness	0.85	0.34	0.37
complexity	0.71	0.71	0.71
coherence	0.79	0.22	0.85
verbosity	0.90	0.96	0.95

Appendix

Table 7: Masking ablation for the label-binarization choice on Llama-3.1-8B-instruct.

Figure 10: Pearson Correlation betweeen different models across eight functions.

Figure 11: Pearson Correlation between different value dimensions across two models.

		Human vs. LLM ( $\kappa$ )			LLM vs. LLM ( $\kappa$ )
Value	$n$	Qwen	GPT-5	Gemini	Qwen–GPT5	Qwen–Gemini	GPT5–Gemini
harmless	50	0.875	0.875	0.875	0.864	1.000	0.864
honesty	50	0.619	0.640	0.665	0.440	0.619	0.880

Appendix

Table 8: Human vs. LLM-judge agreement on 50 annotated Qwen3-8B answers per value (Cohen’s

\kappa

Model	Masked Value	WikiText-2 PPL ( $\downarrow$ )			PTB PPL ( $\downarrow$ )
Model	Masked Value	Base	TopK	Random	Base	TopK	Random
Llama-3.1-8B-Instruct	Harmless	10.14	10.14	10.14	18.36	18.36	18.36
	Helpfulness	10.14	10.14	10.14	18.36	18.36	18.36
	Honesty	10.14	10.14	10.14	18.36	18.36	18.36
Llama-3.2-3B-Instruct	Harmless	16.07	16.07	16.07	30.12	30.12	30.12
	Helpfulness	16.07	16.07	16.07	30.12	30.12	30.12

Appendix

Table 9: Intervention results of value heads vs. random heads across the 3H dimensions—harmlessness, helpfulness, and honesty—on two value-neutral general corpora.

Model	Setting	Harmless	Honesty	Helpfulness	Complexity	Verbosity	Coherence
Llama-3.1-8B	Randomk	0.346 $\pm$ 0.097	0.403 $\pm$ 0.084	0.773 $\pm$ 0.114	0.671 $\pm$ 0.070	0.921 $\pm$ 0.018	0.687 $\pm$ 0.113
	Topk	0.200	0.140	0.340	0.710	0.960	0.220
	$p$	0.014	0.001	0.001	0.860	0.996	0.000
Llama-3.2-3B	Randomk	0.440 $\pm$ 0.121	0.370 $\pm$ 0.022	0.661 $\pm$ 0.053	0.609 $\pm$ 0.065	0.921 $\pm$ 0.016	0.622 $\pm$ 0.079
	Topk	0.340	0.250	0.270	0.660	0.970	0.410
	$p$	0.069	0.000	0.000	0.923	0.999	0.002

Appendix

Table 10: Significance of masking value heads (5 runs). randomk : mean

\pm

std over 5 random-head maskings; topk : masking the deterministic value heads. Lower topk accuracy indicates that the selected value heads are more causally important.

p

: one-sided single-sample

t

-test comparing randomk and topk. Bold:

p<0.05

Model	Alpha	Inter_Head	Harmless $\uparrow$	Honesty $\uparrow$	Helpfulness $\uparrow$	Complexity $\uparrow$	Verbosity $\uparrow$	Coherence $\uparrow$
			Jailbreak	Truthfulqa	HelpSteer2_h	HelpSteer2_c	HelpSteer2_v	HelpSteer2_c
Llama-3.1-8B	0.3	original	0.44	0.61	0.85	0.71	0.90	0.79
	0.3	value inter.	0.45	0.63	0.86	0.72	0.89	0.81
	0.5	original	0.44	0.61	0.85	0.71	0.90	0.79
	0.5	value inter.	0.48	0.61	0.85	0.71	0.90	0.79
	1.0	original	0.44	0.61	0.85	0.71	0.90	0.79

Appendix

Table 11: Positive intervention results (%) on single value dimension across different LLM models on different alpha values.

Model	Inter_Head	Harmless $\uparrow$	Honesty $\uparrow$	Helpfulness $\uparrow$	Complexity $\uparrow$	Verbosity $\uparrow$	Coherence $\uparrow$
Model	Inter_Head	Jailbreak	Truthfulqa	HelpSteer2_h	HelpSteer2_c	HelpSteer2_v	HelpSteer2_c
Llama3.2-3B	original	0.61	0.54	0.82	0.74	0.93	0.77
Llama3.2-3B	value inter.	0.59	0.59	0.83	0.78	0.89	0.75
Qwen3-4B	original	0.44	0.68	0.81	0.79	0.94	0.76
Qwen3-4B	value inter.	0.45	0.70	0.85	0.80	0.96	0.77
Yi-1.5-6B	original	0.16	0.56	0.79	0.76	0.89	0.76

Appendix

Table 12: Positive intervention results (%) on single value dimension across different LLM models with small scale.

Dataset	Model	Inter_Head	Harmless $\uparrow$	Honesty $\uparrow$	Helpfulness $\uparrow$	Complexity $\uparrow$	Verbosity $\uparrow$	Coherence $\uparrow$
BeaverTails	Llama3.1-3B	original	0.61	0.66	0.63	0.58	0.89	0.62
	Llama3.1-3B	value inter.	0.63	0.62	0.64	0.53	0.94	0.66
	Qwen3-4B	original	0.57	0.61	0.67	0.57	0.90	0.63
	Qwen3-4B	value inter.	0.66	0.69	0.75	0.59	0.96	0.74
	Yi-1.5-6B	original	0.40	0.49	0.56	0.63	0.95	0.56
	Yi-1.5-6B	value inter.	0.43	0.55	0.63	0.71	0.98	0.60

Appendix

Table 13: Positive intervention results (%) on multiple value dimension across different LLM models with small scale.

Models	Dataset	Alpha	Inter_Head	Harmless $\uparrow$	Honesty $\uparrow$	Helpfulness $\uparrow$	Complexity $\uparrow$	Verbosity $\uparrow$	Coherence $\uparrow$
Llama-3.1-8B	BeaverTails	0.3	original	0.58	0.62	0.62	0.50	0.94	0.64
		0.3	value inter.	0.58	0.64	0.64	0.52	0.94	0.66
		0.5	original	0.58	0.62	0.62	0.50	0.94	0.64
		0.5	value inter.	0.58	0.63	0.68	0.52	0.94	0.60
		1.0	original	0.58	0.62	0.62	0.50	0.94	0.64
		1.0	value inter.	0.64	0.64	0.68	0.56	0.94	0.66

Appendix

Table 14: Positive intervention results (%) on multiple value dimension across different LLM models and alpha values.

Model	Method	Harmless	Honesty	Helpfulness
Llama3.2-3B	Original	0.465 $\pm$ 0.034	0.570 $\pm$ 0.048	0.582 $\pm$ 0.054
	MODPO	0.437 $\pm$ 0.061	0.541 $\pm$ 0.094	0.580 $\pm$ 0.075
	MOPFT	0.446 $\pm$ 0.065	0.562 $\pm$ 0.090	0.596 $\pm$ 0.063
	Ours	0.545 $\pm$ 0.041	0.621 $\pm$ 0.043	0.652 $\pm$ 0.012
Qwen3-4B	Original	0.633 $\pm$ 0.052	0.656 $\pm$ 0.064	0.675 $\pm$ 0.058
	MODPO	0.624 $\pm$ 0.063	0.677 $\pm$ 0.062	0.674 $\pm$ 0.025

Appendix

Table 15: Multi-objective alignment on Harmless, Honesty, and Helpfulness.

Dataset	Model	Method	Harmless $\uparrow$	Honesty $\uparrow$	Helpfulness $\uparrow$
BeaverTails	Llama3.1-3B	Original	0.61	0.66	0.63
		MODPO	0.55	0.58	0.65
		MOPFT	0.56	0.57	0.65
		Ours	0.63	0.62	0.64
	Llama3.1-8B	Original	0.58	0.62	0.62
		MODPO	0.55	0.58	0.61

Appendix

Table 16: Multi-objective alignment results (%) on Harmless, Honesty, and Helpfulness. “Ours” denotes value intervention via selected attention heads.

Figure 12: Distribution of directional angles between value pairs across different models.

Model	Value Pair	$n$	Mean (°)	Median (°)	Std (°)
Llama3.1-8B	Harmless $\times$ Helpfulness	86	63.81	62.83	12.53
	Harmless $\times$ Honesty	95	93.77	93.53	8.67
	Helpfulness $\times$ Honesty	89	83.51	81.74	10.21
Llama3.2-3B	Harmless $\times$ Helpfulness	50	65.90	66.71	8.44
	Harmless $\times$ Honesty	60	94.44	93.76	6.94
	Helpfulness $\times$ Honesty	53	83.09	83.54	8.74

Appendix

Table 17: Cosine Angles between Value-Pair Direction Vectors on Shared Top-15% Heads

Multi-Objective Language Model Alignment

Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.

See all 96 papers

VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models
Mar 18, 2026Hefei Xu, Le Wu, Yu Wang +4LLM Alignment Multi-Objective Language Model Alignment
Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, Anhui, China · Innovation School of Artificial Intelligence, Hefei University of Technology
Adaptive Multi-Value Control in LLMs via Causal Activation Steering
Sep 24, 2026Payel Bhattacharjee, Ravi TandonLanguage Model Steering Activation Steering
School of Electrical, Computing, and Software Engineering University of Arizona, Tucson, AZ, USA
Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models
Sep 15, 2026Keqing Zhang, Jingyu Chen, Yufan Liu +6LLM Alignment Language Model Bias Evaluation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, 95 Zhongguancun East Road, Beijing, 100190, China. · School of Industry-education Integration, University of Chinese Academy of Sciences, 1 Yanqihu East Road, Huairou, Beijing, 101499, China. · Beihang University, 37 Xueyuan Road, Beijing, 100191, China. +2

Mar 18, 2026J/K move · Enter open · S save

VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models

Hefei Xu, Le Wu, Yu Wang, Min Hou+3

Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology, Hefei, Anhui, China · Innovation School of Artificial Intelligence, Hefei University of Technology

LLM Alignment Multi-Objective Language Model Alignment Model Merging

Open paper AskPDF arXiv

As large language models (LLMs) increasingly shape content generation, interaction, and decision-making across the Web, aligning them with human values has become a central objective in trustworthy AI. This challenge becomes even more pronounced when aligning multiple, potentially conflicting human values. Although recent approaches, such as reward reweighting, prompt-based supervised fine-tuning, and model merging, attempt to tackle multi-value alignment, they still face two major limitations: (1) training separate models for each value combination is prohibitively expensive; (2) value conflicts substantially degrade alignment performance. These limitations make it difficult to achieve favorable trade-offs across diverse human values. To address these challenges, we revisit multi-value alignment from the perspective of value consistency in data and propose VC-soup, a data filtering and parameter merging framework grounded in value-consistent learning. We first design a value consistency metric based on the cosine similarity between the reward-gap vector of each preference pair and an all-ones vector, which quantifies its cross-value coherence. We then filter out low-consistency preference pairs in each value dataset and train on the remaining data to obtain smooth, value-consistent policy models that better preserve linear mode connectivity. Finally, we linearly combine these policies and apply Pareto filtering across values to obtain solutions with balanced multi-value performance. Extensive experiments and theoretical analysis demonstrate that VC-soup effectively mitigates conflicts and consistently outperforms existing multi-value alignment methods.

Abstract

Multi-Objective Language Model Alignment

Explore similar work

VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models

Adaptive Multi-Value Control in LLMs via Causal Activation Steering

Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models