DNAlign: Dynamic Null-Space Safe Alignment for LLMs
Authors: Jisheng Dang, Yushuo Zhao, Dewei Liu, Junfeng Fang, Bimei Wang, Tiantian Rao, Hong Peng, Bin Hu, Tat-Seng Chua
Organizations: School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · National University of Singapore, Singapore 119077 · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China
Ensuring the safe and reliable deployment of large language models (LLMs) remains a fundamental challenge. Existing safety alignment approaches either incur high computational cost or unintentionally disrupt the model's core knowledge, leading to degraded fluency and factual accuracy on benign tasks. This reveals a persistent trade-off between safety and utility. We propose DNAlign, a lightweight alignment framework that integrates control-theoretic optimization with null-space projection. By treating the LLM as a dynamic system, the proposed framework introduces controllable perturbations to steer generation toward safe behavior. A key component is the projection module, which restricts these perturbations to the harmful-related subspace derived from neutral hidden states, thereby preserving general knowledge and response quality. A value function trained on human preference data adaptively optimizes the control signals to align with human safety preferences. Extensive evaluations across multiple LLM backbones demonstrate that our framework consistently reduces harmful outputs while maintaining fluency, coherence, and factual utility. It achieves superior overall performance compared to prior alignment baselines without sacrificing generation diversity. These results indicate that the proposed framework provides an effective and practically deployable solution for safe LLM alignment. Code is available at https://anonymous.4open.science/r/DNAlign.
Figures & tables
Fig. 1: Comparison of LLMs with/without DNAlign alignment. Unedited models forget security-related knowledge, while our proposed framework preserves safe responses.
Fig. 2: The sequential process begins by computing the initial state s0={h0,o0} from the input prompt. It then loads the pre-computed null-space projection matrix P . In the core step, a gradient-optimized control signal ut is projected by P to enforce safety alignment while minimally affecting neutral knowledge. Finally, the projected signal Put updates the system state st+1 , enabling token generation yt .
Fig. 3: End-to-end experimental workflow of the proposed DNAlign framework. This workflow includes four key stages: (1) data preparation (curating neutral prompts from the SHP dataset and harmful prompts from benchmark datasets), (2) offline projection matrix construction (extracting hidden states of neutral prompts to define protected subspaces), (3) online control signal injection (projecting signals into null spaces before modifying LLM states), and (4) performance evaluation (assessing both safety alignment and neutral knowledge preservation via multi-metric validation).
Backbone
Dataset
Model
Diversity
Coherence
Average Reward
Comprehensive Quality
Vicuna-7B [ 33 ]
HH-RLHF
Base
0.8672
–
0.3991
–
0.1410
–
5.3600
–
CD [ 46 ]
0.8810
–
0.4081
–
0.8440
↑
5.6800
↑
Static RE [ 36 ]
0.8941
↑
0.3813
–
0.9130
↑
5.0500
↓
RE-Control [ 34 ]
0.9007
↑
0.3887
–
0.9614
↑
5.9400
↑
DNAlign (Ours)
0.9026
↑
0.4325
↑
1.2258
↑
6.1600
↑
SHP
Base
0.8833
–
0.3762
–
0.3913
–
6.3300
–
TABLE I: Comprehensive performance comparison on the HH-RLHF and SHP datasets.
Fig. 4: The influence of step size and epochs at test-time.
Fig. 5: Variation of inference time across three models with training batches.
Fig. 6: Research on sample parameters of projection matrix.
Fig. 7: Generalization Performance of DNAlign on Harmful Datasets. Comparison of DNAlign against the base model (Qwen-7B) and RE-Control across three distinct harmful datasets: RealToxicityPrompts, ToxiGen, and Categorical-HarmfulQA. DNAlign achieves superior performance in safety alignment (average reward), notably increasing it by 137.1% on RealToxicityPrompts and 229.4% on ToxiGen compared to the base model. The results confirm DNAlign’s strong generalization ability, effectively balancing significant safety improvements with competitive diversity and coherence across diverse harmful content distributions.
Fig. 8: Performance on Harmless Datasets and Core Capability Preservation. Evaluation of DNAlign compared to the base model and RE-Control across six diverse harmless datasets, assessing core LLM capabilities. DNAlign successfully maintains competitive diversity (0.8464 to 0.8836) and coherence (0.4077 to 0.5692) scores close to the base model, demonstrating minimal degradation of core capabilities. Simultaneously, DNAlign achieves a superior average reward, outperforming RE-Control and often showing significant safety improvement (up to 81.9% gain in average reward over the base model on specific datasets). This confirms that the null-space projection effectively balances safety alignment and utility.
Safety alignment of large language models (LLMs) typically depends on high-quality supervision data, such as safe demonstrations or preference pairs. However, in real-world deployment, emerging safety requirements are often specified as natural-language policies, while corresponding supervision data may be costly, delayed, or unavailable. This creates a mismatch between rapidly evolving safety policies and conventional data-driven alignment methods. To address this, we propose PolicyAlign, a simple yet effective framework for directly aligning LLMs with safety policies. Given a safety policy, PolicyAlign first synthesizes policy-violating instructions and then performs on-policy self-distillation to internalize policy-guided behavior. To improve training stability and data efficiency, we further introduce Policy-Sensitive Filtering, which selects instructions where the policy induces the largest behavioral shift. Experiments across multiple models show that PolicyAlign consistently improves safety while maintaining low over-refusal and preserving general capabilities. PolicyAlign also generalizes to medical, legal, and financial safety scenarios, highlighting its potential as a scalable and maintainable approach to policy-based LLM safety alignment. The code is released at https://github.com/Qwen-Applications/PolicyAlign.
Chang Wu, Junfeng Fang, Houcheng Jiang +5
Qwen Large Model Application Team, Alibaba · National University of Singapore · Zhongguancun Academy
We introduce a training-free safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning or reinforcement learning from human feedback. Our main idea is to provide a robust plug-and-play approach to prevent shadow alignment when models are adapted to downstream tasks. Specifically, we leverage knowledge distillation to extract alignment signals from well-aligned LLMs and inject them into shadow-aligned models via model fusion, enabling plug-and-play alignment correction. In our methodology, we employ delta debugging to identify the critical components of knowledge necessary for effective distillation. On the harmful question dataset, our method significantly enhances the average defense success rate by approximately 14.42%, reaching as high as 51.39% across 17 influenced LLMs, without compromising performance. Our code is available at https://github.com/NWULIST/DAPA.
Haozheng Luo, Jiahao Yu, Wenxin Zhang +9
Northwestern University · New York University Abu Dhabi · Stanford University +3
Fine-tuning well-aligned large language models (LLMs) on new domains often degrades their safety alignment, even when using benign datasets. Existing safety alignment techniques primarily focus on pretraining, leaving fine-tuned models vulnerable to behavioral shifts. In this work, we introduce safety token regularization (STR), a lightweight method designed to preserve safety properties during fine-tuning. Our approach identifies salient tokens from rejection templates of well-aligned models and constrains their associated logits during training, preventing the loss of critical safety behaviors. Unlike reinforcement learning or preference optimization methods, STR requires minimal additional computation and seamlessly integrates with parameter-efficient fine-tuning techniques such as LoRA. Comprehensive experiments demonstrate that our approach achieves safety performance on par with state-of-the-art methods, while preserving task-specific utility and requiring minimal implementation overhead. Furthermore, we show that safety token regularization enhances training stability and overall performance beyond safety considerations alone. This work offers a practical and readily deployable strategy for continual safety alignment in fine-tuned LLMs.
Thong Bach, Truyen Tran
Applied Artificial Intelligence Initiative (A2I2), Deakin University