Aligning Large Language Models (LLMs) with nuanced human values remains a critical challenge, as existing methods like Reinforcement Learning from Human Feedback (RLHF) often handle only coarse-grained attributes. In practice, fine-tuning LLMs on task-specific datasets to optimize value alignment inevitably incurs an alignment tax: the model's pre-calibrated value system drifts significantly due to latent bias absorption from training data, while the fine-tuning process also causes severe hallucinations and semantic information loss in generated responses. To address this, we propose VISA (Value Injection via Shielded Adaptation), a closed-loop framework designed to navigate this trade-off. VISA's architecture features a high-precision value detector, a semantic-to-value translator, and a core value-rewriter. The value-rewriter is trained via Group Relative Policy Optimization (GRPO) with a composite reward function that simultaneously optimizes for fine-grained value precision, and the preservation of semantic integrity. By learning an optimal policy to balance these competing objectives, VISA effectively mitigates the alignment tax while staying loyal to the original knowledge. Our experiments demonstrate that this approach enables precise control over a model's value expression while maintaining its factual consistency and general capabilities, significantly outperforming both standard fine-tuning methods and prompting-based baselines, including GPT-4o.
Figures & tables
Figure 1. VISA navigates the trade-off between knowledge preservation and value alignment, finding a safe path to personalized models.
Figure 2. The phenomenon of Value Drift (the unintended shift of models’ foundational values after knowledge fine-tuning). When a base model is fine-tuned on a knowledge-centric dataset, its value shifts sharply in the four-dimensional scale, demonstrating an undesirable value drift. Value dimensions are scored using methodology adapted from Zhang et al. ( Zhang et al., 2025b ) .
Figure 3. The VISA pipeline is designed to decouple knowledge preservation from value alignment. The User Workflow (top) shows the inference-time process: a user’s textual instruction is interpreted by the Translator into a latent Value Shift Vector ( ΔV ). Concurrently, the Detector analyzes the Original Response to extract its intrinsic Original Value Vector. These vectors are combined to form a precise Target Value. The core Rewriter model then conditions on the original response and this target vector to produce a new, value-aligned output. The Training Process (bottom) details how the Rewriter is optimized using GRPO. It learns to maximize a dual-objective reward signal, combining Value Reward (cosine similarity to the target value vector) and Consistency Reward (semantic entailment with the original response), thereby learning to inject values without hallucinating or losing factual information.
Figure 4. Human evaluation on value rewriting quality and precision. (a) Our model outperforms all baselines in pairwise preference comparisons. (b) In terms of value identification consistency, our model achieves the highest average match score (7.60/10) with the lowest variance.
Method
Model
consistency
consistency(fwd)
consistency(bwd)
value(l2_distance)
value(cos_sim)
mean ↑
std ↓
mean ↑
std ↓
mean ↑
std ↓
mean ↓
std ↓
mean ↑
std ↓
Simple-Prompt
GPT-4o-Mini
0.8406
0.1496
0.8523
0.1599
0.8172
0.1979
0.7986
0.3011
0.6935
0.3167
GPT-4o
0.7831
0.1777
0.8171
0.1883
0.7151
0.2666
0.7717
0.2877
0.7089
0.3079
Gemini-3-Pro
0.6128
0.2174
0.6230
0.2385
0.5924
0.3494
0.7095
0.2551
0.7459
0.2870
Rule-Based-Instruct
GPT-4o-mini
0.7652
0.1721
0.7564
0.1994
0.7829
0.2042
0.8033
0.3040
0.6977
0.3138
GPT-4o
0.6792
0.2349
0.6971
0.2563
0.6434
0.3078
0.7719
0.2893
0.7214
0.3069
Table 1. Comparison of different methods on raw cosine similarity and quantized Euclidean distance.
Figure 5. Qualitative comparison of VISA and GPT-4o on a value rewriting task. VISA successfully injects the target values while maintaining high knowledge consistency, whereas the prompted GPT-4o achieves lower value cosine consistency and deviates from the original information. Refer to Section 4.4 for detailed analysis.
Figure 6. Performance comparison of alignment methods. (a) Semantic Consistency ( ↑ ) and (b) Value L2 Distance ( ↓ ) relative to the target vector v∗ . (c) Joint Success Rate (JSR) across different model scales. Our method consistently outperforms baselines (SFT, DPO, SimPO) in balancing value injection and semantic preservation.
Figure 7. Overview of the Adaptive Value Search pipeline. The framework operates as a bi-level loop: the Inner Loop evaluates the potential of sampled value vectors via mock updates, while the Outer Loop refines the distribution parameters based on high-advantage samples. This iterative process steers the search from broad exploration ( t=0 ) to precise convergence ( t=T ).
Method
Value Drift ( ↓ )
MMLU Math ( ↑ )
Model: Qwen3-0.6B
Base
0.0000
0.3645
SFT
0.1191
0.3807
Ours (VISA)
0.0775 (-34.9%)
0.3855 (+1.3%)
Model: Qwen3-8B
Base
0.0000
0.5224
Table 2. Performance comparison on Domain Fine-tuning. We report the Value Drift (deviation from original alignment, lower is better.) and Domain Score (MMLU-Math accuracy, higher is better).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8. Distribution of Relative Value Shifts ( Δv ) across Dimensions. The violin plot illustrates the density of value changes in the VCR-45K dataset. It highlights (1) Multi-Scale Editing Granularity , covering both subtle shifts ( ±0.5 ) and radical reconstruction ( ≥1.5 ), and (2) Orthogonal Disentanglement , indicated by the significant expansion at Δv=0 , showing that non-target dimensions remain largely strictly preserved.
Figure 9. Human evaluation on value rewriting quality and precision. (a) Our model outperforms all baselines in pairwise preference comparisons with a 57.0% win rate. (b) In terms of value identification consistency, our model achieves the highest average match score (7.60/10) with the lowest variance, demonstrating its effectiveness in precise value injection and instruction following.
Table 3. Reference guide for Schwartz Value Dimensions used in Human Annotation.
Hyperparameter
Value
Phase 1: SFT Cold Start
Optimization Stage
Full Parameter Fine-tuning
DeepSpeed Config
ZeRO-3 (FP8)
Learning Rate
1.0e-5
LR Scheduler
Cosine
Warmup Ratio
0.1
Appendix
Table 4. Hyperparameter settings for SFT and GRPO phases. The table details the optimization configuration, data processing parameters, and generation strategies used in our experiments.
Consist.
Consist. Fwd
Consist. Bwd
Val L2
Jnt SR
Univ
Benev
Trad
Conf
Sec
Pow
Ach
Hed
Stim
SelfD
MAD
mean
std
mean
std
mean
std
mean
std
Llama3.1-8B
vanilla
0.4194
0.2013
0.334
0.2341
0.5901
0.3221
0.8156
0.2391
0.3671
0.2439
0.2298
0.1515
0.2275
0.2309
0.1002
0.2292
0.1309
0.2039
0.2377
0.1986
sft
0.1757
0.1824
0.1799
0.2164
0.1675
0.2081
0.8761
0.2871
0.0817
0.2661
0.2383
0.1454
0.2190
0.2493
0.0939
0.2635
0.1347
0.2039
0.2578
0.2072
dpo
0.4019
0.2015
0.3312
0.2360
0.5433
0.3171
0.8138
0.2389
0.3395
0.2507
0.2295
0.1470
0.2243
0.2269
0.0996
0.2260
0.1349
0.2052
0.2405
0.1985
simpo
0.4098
0.2012
0.3359
0.2309
0.5575
0.3173
0.8127
0.2399
0.3510
0.2487
0.2278
0.1469
0.2208
0.2290
0.1005
0.2325
0.1364
0.2036
0.2375
0.1984
grpo
0.8195
0.1067
0.7676
0.1347
0.9233
0.1323
0.7421
0.2225
0.6502
0.2443
0.1713
0.1192
0.2132
0.1986
0.0903
0.2197
0.1068
0.1737
0.2779
0.1815
Appendix
Table 5. Consistency and Value Alignment Results. Consistency metrics (mean ± std), value L2 distance, joint success rate, and Schwartz value dimensions across models and training methods. For each model block, we highlight the best and worst performers on key trade-off metrics.
Aligning large language models (LLMs) with human values typically relies on post-training or inference-time steering that directly manipulates the backbone's parameters or representation space. However, a critical gap exists: the model's residual stream is highly dynamic, in which values exist as fragile, low-dimensional properties, inherently incompatible with the stability required for consistent value expression. In this paper, we propose the Stable Value Guidance Transformer (SVGT), which addresses this gap through an independent value module incorporating two key designs: (1) independent value modeling, maintaining normative representations in a dedicated value space isolated from the backbone, and (2) explicit behavioral guidance, transducing these stable signals into learnable latent Bridge Tokens. These tokens serve as dynamic value anchors to explicitly steer the generative trajectory, ensuring robust adherence across diverse contexts without disrupting the backbone's internal representations. Experiments across multiple backbones and safety benchmarks show that SVGT generally reduces harmful scores by over 70% while maintaining generation fluency, demonstrating the efficacy of architecturally grounded value modeling. Our code is available at https://github.com/Clervils/SVGT.git.
Wenhao Chen, Sirui Sun, Shengyuan Bai +1
School of Electronics Engineering and Computer Science, Peking University · Yuanpei College, Peking University · State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
Fine-tuning well-aligned large language models (LLMs) on new domains often degrades their safety alignment, even when using benign datasets. Existing safety alignment techniques primarily focus on pretraining, leaving fine-tuned models vulnerable to behavioral shifts. In this work, we introduce safety token regularization (STR), a lightweight method designed to preserve safety properties during fine-tuning. Our approach identifies salient tokens from rejection templates of well-aligned models and constrains their associated logits during training, preventing the loss of critical safety behaviors. Unlike reinforcement learning or preference optimization methods, STR requires minimal additional computation and seamlessly integrates with parameter-efficient fine-tuning techniques such as LoRA. Comprehensive experiments demonstrate that our approach achieves safety performance on par with state-of-the-art methods, while preserving task-specific utility and requiring minimal implementation overhead. Furthermore, we show that safety token regularization enhances training stability and overall performance beyond safety considerations alone. This work offers a practical and readily deployable strategy for continual safety alignment in fine-tuned LLMs.
Thong Bach, Truyen Tran
Applied Artificial Intelligence Initiative (A2I2), Deakin University
Ensuring the safe and reliable deployment of large language models (LLMs) remains a fundamental challenge. Existing safety alignment approaches either incur high computational cost or unintentionally disrupt the model's core knowledge, leading to degraded fluency and factual accuracy on benign tasks. This reveals a persistent trade-off between safety and utility. We propose DNAlign, a lightweight alignment framework that integrates control-theoretic optimization with null-space projection. By treating the LLM as a dynamic system, the proposed framework introduces controllable perturbations to steer generation toward safe behavior. A key component is the projection module, which restricts these perturbations to the harmful-related subspace derived from neutral hidden states, thereby preserving general knowledge and response quality. A value function trained on human preference data adaptively optimizes the control signals to align with human safety preferences. Extensive evaluations across multiple LLM backbones demonstrate that our framework consistently reduces harmful outputs while maintaining fluency, coherence, and factual utility. It achieves superior overall performance compared to prior alignment baselines without sacrificing generation diversity. These results indicate that the proposed framework provides an effective and practically deployable solution for safe LLM alignment. Code is available at https://anonymous.4open.science/r/DNAlign.
Jisheng Dang, Yushuo Zhao, Dewei Liu +6
School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · National University of Singapore, Singapore 119077 · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China