Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models

Utkarsh Agarwal; Monojit Choudhury

cs.CLSep 17, 2026

Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models

Authors: Utkarsh Agarwal, Monojit Choudhury

Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE.

Abstract

Large Language Models (LLMs) are increasingly deployed in applications that must weigh clashing moral values, yet even strong models exhibit hidden biases and brittle instruction-following across languages. We introduce a 12,000-instance dataset of two-option dilemmas covering pairwise three value conflicts: Honesty vs. Justice, Justice vs. Autonomy, and Autonomy vs. Honesty, along with their translations into Hindi, Arabic, Spanish, and Chinese, to probe cross-lingual behavior. Benchmarking on GPT-5-mini reveals that it consistently favors Honesty over Autonomy across all five languages when no policy is given. The Llama-3.2-1/3B models exhibit strong first-option bias; however, both plain fine-tuning and Direct Preference Optimization fine-tuning effectively remove this bias, increasing accuracy to greater than 98%. In order to decouple the effect of learning correlations in the dataset from abstract values, we propose a task vector transfer based experiment where after computing the task vectors for a direction of value preference we orthogonalize it with respect to the general instruction following vector. Our experiment shows that this method is effective in isolating the direction of the specific value preference that can successfully be used to conduct task arithmetic to obtain a model with the opposite stance.

Figures & tables

Figure 1: An Ethical Dilemma scenario where an agent is present and is faced with a question and two options. Both options are virtuous and there isn’t a right answer without an associated ethical stance.

	Task_AB	Task_BC	Task_CA
English	56.8%	69.5%	30.0%
Hindi	64.5%	73.5%	27.2%
Spanish	58.8%	68.5%	29.5%
Arabic	60.5%	67.5%	28.0%
Chinese	62.0%	71.5%	29.2%

Table 1: Baseline preference of GPT–5-mini . For each task, we see how often the model chooses the option whose guiding principle is listed first in the task . For

Task\_ij

, we report here the percentage of times the model chose the option preferring value

i

over value

j

	Task_AB		Task_BC		Task_CA
	S1	S2	S1	S2	S1	S2
English	85.8	72.8	98.8	99.2	98.0	95.0
Hindi	61.8	57.0	95.5	88.2	75.0	90.2
Spanish	88.0	70.0	99.2	99.5	98.8	93.5
Arabic	79.8	72.2	99.5	99.8	94.2	92.5
Chinese	92.2	91.8	99.8	97.2	97.0	96.5

Table 2: Prompt‑steering on GPT–5-mini “Stance‑1” instructs the model to follow the default order (e.g.

v_{A}\succ v_{B}

for

Task\_AB

); “Stance‑2” explicitly instructs the reverse order (

v_{B}\succ v_{A}

). Numbers in the table show the accuracy of the model.

Figure 2: SFT Accuracy: Accuracy of models on the gold test data for each language and task trained on corresponding training data. Task AB implies that the model’s task was to prefer value A over value B. A, B, C are as defined in § 3.1 .

Figure 3: DPO Accuracy: Accuracy of models trained using DPO on the gold test set. Task AB implies that the model’s task was to prefer value A over value B. A, B, C are as defined in § 3.1 .

Figure 4: Task-vector transfer efficiency. Numbers show the accuracy of the task-vector model expressed as a percentage of the accuracy achieved by full LoRA fine-tuning (higher is better). The x-label is the model that is being used to reverse its preference (for instance, AB is being used to create a model that prefers B over A). In a box, BC_CB indicates that BC and CB models were used to create the instruction vector. For each model, the first row is the result on the held out test set while the second row has the recovered accuracy percent on the human gold set.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: SFT Accuracy: Accuracy of models on the test data for each language and task trained on corresponding training data. Task AB implies that the model’s task was to prefer value A over value B.

Figure 6: DPO Accuracy: Accuracy of models trained using DPO. Each language-task pair was trained with three different seeds. Task AB implies that the model’s task was to prefer value A over value B.

Figure 7: Scatter plot of the optimum

\gamma_{1}(\gamma_{instr})

and

\gamma_{2}(\gamma_{pref})

. Only the top 80 percent of the points are shown, scaled by the test accuracy and with small random jitters to prevent overlapping and demonstrate the number of points there.

Figure 8: Task-vector transfer efficiency. Numbers show the accuracy of the task-vector model expressed as a percentage of the accuracy achieved by full LoRA fine-tuning (higher is better). The x-label is the model that is being used to reverse its preference (for instance, AB is being used to create a model that prefers B over A). In a box, BC_CB indicates that BC and CB models were used to create the instruction vector. The better performing combination is placed above the other one.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

Same Values, Different Languages? From Multilingual Probing to Steering LLMs Toward Chinese Social Values
Sep 8, 2026Yuemei Xu, Kexin Xu, Jian Zhou +3Multilingual Language Model Evaluation Cultural Alignment
School of Information Science and Technology, Beijing Foreign Studies University · Beijing Academy of Artificial Intelligence · The State Key Lab of Software Development Environment, Beihang University
Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models
Feb 22, 2026Seong Hah Cho, Junyi Li, Anna LeshinskayaLLM Alignment Moral Reasoning in Language Models
Independent · Department of Cognitive Sciences, UC Irvine · Department of Cognitive Sciences, UC Irvine & AI Objectives Institute
LKValues: Aligning Large Language Models with Sri Lankan Societal Values
Jul 22, 2026Nethmi Muthugala, Supryadi, Surangika Ranathunga +7Multilingual Language Model Evaluation LLM Alignment
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, Tianjin, China · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand · Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka +2

Sep 8, 2026J/K move · Enter open · S save

Same Values, Different Languages? From Multilingual Probing to Steering LLMs Toward Chinese Social Values

Yuemei Xu, Kexin Xu, Jian Zhou, Haoyu Lu+2

School of Information Science and Technology, Beijing Foreign Studies University · Beijing Academy of Artificial Intelligence · The State Key Lab of Software Development Environment, Beihang University

Multilingual Language Model Evaluation Cultural Alignment Cross-Lingual Consistency in Language Models Activation Steering

Open paper AskPDF arXiv

As Large Language Models (LLMs) are increasingly integrated into human society, aligning them with pluralistic social values has become a critical priority. However, whether LLMs exhibit consistent value preferences across languages remains underexplored, particularly for culturally grounded values, which are more abstract and difficult to evaluate and align than safety-centric principles. We investigate this issue through Chinese Social Values (CSV), a value system rooted in Chinese culture and comprising

12

dimensions across national, societal, and personal levels. We construct C-Voices, the first comprehensive multilingual contrastive probe dataset for CSV, with 86,400 dilemma-based instances in six languages, each pairing a CSV-aligned action with a value-conflicting alternative. Building on the contrastive probes of C-Voices, we then propose a fine-tuning-free value vector steering method that derives value directions from hidden-state discrepancies and selectively intervenes on value-sensitive layers during inference. Experiments on six languages show that CSV-oriented preferences are model-dependent and language-sensitive, with the same dilemma eliciting divergent responses across languages. Our method achieves effective CSV steering, supports cross-lingual transfer of value vectors, and generalizes to existing FLAMES and ValuePrism.

Abstract

LLM Alignment

Explore similar work

Same Values, Different Languages? From Multilingual Probing to Steering LLMs Toward Chinese Social Values

Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models

LKValues: Aligning Large Language Models with Sri Lankan Societal Values