VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment

Jiawei Chen; Tianzhuo Yang; Guoxi Zhang; Jiaming Ji; Yaodong Yang; Juntao Dai

cs.AIMar 5, 2026

VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment

Authors: Jiawei Chen, Tianzhuo Yang, Guoxi Zhang, Jiaming Ji, Yaodong Yang, Juntao Dai

Organizations: Peking University, Beijing, China

Abstract

Aligning Large Language Models (LLMs) with nuanced human values remains a critical challenge, as existing methods like Reinforcement Learning from Human Feedback (RLHF) often handle only coarse-grained attributes. In practice, fine-tuning LLMs on task-specific datasets to optimize value alignment inevitably incurs an alignment tax: the model's pre-calibrated value system drifts significantly due to latent bias absorption from training data, while the fine-tuning process also causes severe hallucinations and semantic information loss in generated responses. To address this, we propose VISA (Value Injection via Shielded Adaptation), a closed-loop framework designed to navigate this trade-off. VISA's architecture features a high-precision value detector, a semantic-to-value translator, and a core value-rewriter. The value-rewriter is trained via Group Relative Policy Optimization (GRPO) with a composite reward function that simultaneously optimizes for fine-grained value precision, and the preservation of semantic integrity. By learning an optimal policy to balance these competing objectives, VISA effectively mitigates the alignment tax while staying loyal to the original knowledge. Our experiments demonstrate that this approach enables precise control over a model's value expression while maintaining its factual consistency and general capabilities, significantly outperforming both standard fine-tuning methods and prompting-based baselines, including GPT-4o.

Figures & tables

Figure 1. VISA navigates the trade-off between knowledge preservation and value alignment, finding a safe path to personalized models.

Figure 2. The phenomenon of Value Drift (the unintended shift of models’ foundational values after knowledge fine-tuning). When a base model is fine-tuned on a knowledge-centric dataset, its value shifts sharply in the four-dimensional scale, demonstrating an undesirable value drift. Value dimensions are scored using methodology adapted from Zhang et al. ( Zhang et al., 2025b ) .

Figure 3. The VISA pipeline is designed to decouple knowledge preservation from value alignment. The User Workflow (top) shows the inference-time process: a user’s textual instruction is interpreted by the Translator into a latent Value Shift Vector ( $\Delta V$ ). Concurrently, the Detector analyzes the Original Response to extract its intrinsic Original Value Vector. These vectors are combined to form a precise Target Value. The core Rewriter model then conditions on the original response and this target vector to produce a new, value-aligned output. The Training Process (bottom) details how the Rewriter is optimized using GRPO. It learns to maximize a dual-objective reward signal, combining Value Reward (cosine similarity to the target value vector) and Consistency Reward (semantic entailment with the original response), thereby learning to inject values without hallucinating or losing factual information.

Figure 4. Human evaluation on value rewriting quality and precision. (a) Our model outperforms all baselines in pairwise preference comparisons. (b) In terms of value identification consistency, our model achieves the highest average match score (7.60/10) with the lowest variance.

Method	Model	consistency		consistency(fwd)		consistency(bwd)		value(l2_distance)		value(cos_sim)
		mean ↑	std ↓	mean ↑	std ↓	mean ↑	std ↓	mean ↓	std ↓	mean ↑	std ↓
Simple-Prompt	GPT-4o-Mini	0.8406	0.1496	0.8523	0.1599	0.8172	0.1979	0.7986	0.3011	0.6935	0.3167
	GPT-4o	0.7831	0.1777	0.8171	0.1883	0.7151	0.2666	0.7717	0.2877	0.7089	0.3079
	Gemini-3-Pro	0.6128	0.2174	0.6230	0.2385	0.5924	0.3494	0.7095	0.2551	0.7459	0.2870
Rule-Based-Instruct	GPT-4o-mini	0.7652	0.1721	0.7564	0.1994	0.7829	0.2042	0.8033	0.3040	0.6977	0.3138
	GPT-4o	0.6792	0.2349	0.6971	0.2563	0.6434	0.3078	0.7719	0.2893	0.7214	0.3069

Table 1. Comparison of different methods on raw cosine similarity and quantized Euclidean distance.

Figure 5. Qualitative comparison of VISA and GPT-4o on a value rewriting task. VISA successfully injects the target values while maintaining high knowledge consistency, whereas the prompted GPT-4o achieves lower value cosine consistency and deviates from the original information. Refer to Section 4.4 for detailed analysis.

Figure 6. Performance comparison of alignment methods. (a) Semantic Consistency (

\uparrow

) and (b) Value L2 Distance (

\downarrow

) relative to the target vector

v^{*}

. (c) Joint Success Rate (JSR) across different model scales. Our method consistently outperforms baselines (SFT, DPO, SimPO) in balancing value injection and semantic preservation.

Figure 7. Overview of the Adaptive Value Search pipeline. The framework operates as a bi-level loop: the Inner Loop evaluates the potential of sampled value vectors via mock updates, while the Outer Loop refines the distribution parameters based on high-advantage samples. This iterative process steers the search from broad exploration ( $t=0$ ) to precise convergence ( $t=T$ ).

Model: Qwen3-0.6B
Method	Value Drift ( $\downarrow$ )	MMLU Math ( $\uparrow$ )
Base	0.0000	0.3645
SFT	0.1191	0.3807
Ours (VISA)	0.0775 (-34.9%)	0.3855 (+1.3%)
Model: Qwen3-8B
Base	0.0000	0.5224

Table 2. Performance comparison on Domain Fine-tuning. We report the Value Drift (deviation from original alignment, lower is better.) and Domain Score (MMLU-Math accuracy, higher is better).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8. Distribution of Relative Value Shifts (

\Delta v

) across Dimensions. The violin plot illustrates the density of value changes in the VCR-45K dataset. It highlights (1) Multi-Scale Editing Granularity , covering both subtle shifts (

\pm 0.5

) and radical reconstruction (

\geq 1.5

), and (2) Orthogonal Disentanglement , indicated by the significant expansion at

\Delta v=0

, showing that non-target dimensions remain largely strictly preserved.

Figure 9. Human evaluation on value rewriting quality and precision. (a) Our model outperforms all baselines in pairwise preference comparisons with a 57.0% win rate. (b) In terms of value identification consistency, our model achieves the highest average match score (7.60/10) with the lowest variance, demonstrating its effectiveness in precise value injection and instruction following.

Dimension	Core Definition & Typical Signals	Keywords
Self-Direction	Independence & Creativity. Encourages decision-making, innovation, “be yourself.”	Freedom, Curiosity
Stimulation	Novelty & Challenge. Recommends exciting activities, risk-taking, variety.	Adventure, Daring
Hedonism	Pleasure & Enjoyment. Focuses on sensory gratification, comfort, avoiding pain.	Joy, Comfort
Achievement	Success & Competence. Emphasizes efficiency, ambition, result-oriented.	Success, Capable
Power	Status & Control. Focuses on prestige, wealth, influence, authority.	Authority, Wealth
Security	Stability & Order. Refuses dangerous requests, emphasizes safety/privacy.	Safety, Health

Appendix

Table 3. Reference guide for Schwartz Value Dimensions used in Human Annotation.

Phase 1: SFT Cold Start
Hyperparameter	Value
Optimization Stage	Full Parameter Fine-tuning
DeepSpeed Config	ZeRO-3 (FP8)
Learning Rate	1.0e-5
LR Scheduler	Cosine
Warmup Ratio	0.1

Appendix

Table 4. Hyperparameter settings for SFT and GRPO phases. The table details the optimization configuration, data processing parameters, and generation strategies used in our experiments.

		Consist.		Consist. Fwd		Consist. Bwd		Val L2		Jnt SR	Univ	Benev	Trad	Conf	Sec	Pow	Ach	Hed	Stim	SelfD	MAD
		mean	std	mean	std	mean	std	mean	std
Llama3.1-8B	vanilla	0.4194	0.2013	0.334	0.2341	0.5901	0.3221	0.8156	0.2391	0.3671	0.2439	0.2298	0.1515	0.2275	0.2309	0.1002	0.2292	0.1309	0.2039	0.2377	0.1986
	sft	0.1757	0.1824	0.1799	0.2164	0.1675	0.2081	0.8761	0.2871	0.0817	0.2661	0.2383	0.1454	0.2190	0.2493	0.0939	0.2635	0.1347	0.2039	0.2578	0.2072
	dpo	0.4019	0.2015	0.3312	0.2360	0.5433	0.3171	0.8138	0.2389	0.3395	0.2507	0.2295	0.1470	0.2243	0.2269	0.0996	0.2260	0.1349	0.2052	0.2405	0.1985
	simpo	0.4098	0.2012	0.3359	0.2309	0.5575	0.3173	0.8127	0.2399	0.3510	0.2487	0.2278	0.1469	0.2208	0.2290	0.1005	0.2325	0.1364	0.2036	0.2375	0.1984
	grpo	0.8195	0.1067	0.7676	0.1347	0.9233	0.1323	0.7421	0.2225	0.6502	0.2443	0.1713	0.1192	0.2132	0.1986	0.0903	0.2197	0.1068	0.1737	0.2779	0.1815

Appendix

Table 5. Consistency and Value Alignment Results. Consistency metrics (mean

\pm

std), value L2 distance, joint success rate, and Schwartz value dimensions across models and training methods. For each model block, we highlight the best and worst performers on key trade-off metrics.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

Toward Stable Value Alignment: Introducing Independent Modules for Consistent Value Guidance
May 12, 2026Wenhao Chen, Sirui Sun, Shengyuan Bai +1LLM Alignment LLM Safety Alignment
School of Electronics Engineering and Computer Science, Peking University · Yuanpei College, Peking University · State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
Guardrails in Logit Space: Safety Token Regularization for LLM Alignment
Apr 19, 2026Thong Bach, Truyen TranLLM Fine-Tuning LLM Safety Alignment
Applied Artificial Intelligence Initiative (A2I2), Deakin University
DNAlign: Dynamic Null-Space Safe Alignment for LLMs
Oct 2, 2026Jisheng Dang, Yushuo Zhao, Dewei Liu +6LLM Alignment LLM Safety Alignment
School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · National University of Singapore, Singapore 119077 · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China

May 12, 2026J/K move · Enter open · S save

Toward Stable Value Alignment: Introducing Independent Modules for Consistent Value Guidance

Wenhao Chen, Sirui Sun, Shengyuan Bai, Guojie Song

School of Electronics Engineering and Computer Science, Peking University · Yuanpei College, Peking University · State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University

LLM Alignment LLM Safety Alignment Value Alignment Text Generation

Open paper AskPDF arXiv

Aligning large language models (LLMs) with human values typically relies on post-training or inference-time steering that directly manipulates the backbone's parameters or representation space. However, a critical gap exists: the model's residual stream is highly dynamic, in which values exist as fragile, low-dimensional properties, inherently incompatible with the stability required for consistent value expression. In this paper, we propose the Stable Value Guidance Transformer (SVGT), which addresses this gap through an independent value module incorporating two key designs: (1) independent value modeling, maintaining normative representations in a dedicated value space isolated from the backbone, and (2) explicit behavioral guidance, transducing these stable signals into learnable latent Bridge Tokens. These tokens serve as dynamic value anchors to explicitly steer the generative trajectory, ensuring robust adherence across diverse contexts without disrupting the backbone's internal representations. Experiments across multiple backbones and safety benchmarks show that SVGT generally reduces harmful scores by over 70% while maintaining generation fluency, demonstrating the efficacy of architecturally grounded value modeling. Our code is available at https://github.com/Clervils/SVGT.git.

Abstract

LLM Alignment

Explore similar work

Toward Stable Value Alignment: Introducing Independent Modules for Consistent Value Guidance

Guardrails in Logit Space: Safety Token Regularization for LLM Alignment

DNAlign: Dynamic Null-Space Safe Alignment for LLMs